ClaudeBot

by Anthropic

Training crawlers · Model data

Collects public web content that may be used to train Anthropic's Claude models.

Crawlers (or robots.txt controls) that collect public content used to train AI models.

Control it with robots.txt

Well-behaved crawlers read robots.txt before crawling. To block ClaudeBot on your whole site:

robots.txt

User-agent: ClaudeBot
Disallow: /

Blocking AI search and answer crawlers can also remove your pages from AI answers and citations. Weigh traffic from AI assistants before blocking them.

Other Anthropic crawlers

Claude-User

Fetches a page when a Claude user asks a question that needs it.

Claude-SearchBot

Crawls the web to improve the quality of search results returned by Claude.

anthropic-ai

Older Anthropic user-agent token that still appears in many robots.txt files.

← All crawlers