Claude-SearchBot

by Anthropic

Search indexes · Discovery

Crawls the web to improve the quality of search results returned by Claude.

Crawlers that build the search indexes used by search engines and AI search features.

Control it with robots.txt

Well-behaved crawlers read robots.txt before crawling. To block Claude-SearchBot on your whole site:

robots.txt

User-agent: Claude-SearchBot
Disallow: /

Blocking AI search and answer crawlers can also remove your pages from AI answers and citations. Weigh traffic from AI assistants before blocking them.

Other Anthropic crawlers

Claude-User

Fetches a page when a Claude user asks a question that needs it.

ClaudeBot

Collects public web content that may be used to train Anthropic's Claude models.

anthropic-ai

Older Anthropic user-agent token that still appears in many robots.txt files.

← All crawlers