anthropic-ai
by Anthropic
Other AI bots · AI crawler
Older Anthropic user-agent token that still appears in many robots.txt files.
Data-extraction and dataset crawlers whose content often ends up in AI products.
Control it with robots.txt
Well-behaved crawlers read robots.txt before crawling. To block anthropic-ai on your whole site:
robots.txt
User-agent: anthropic-ai Disallow: /
Blocking AI search and answer crawlers can also remove your pages from AI answers and citations. Weigh traffic from AI assistants before blocking them.