AI crawlers, search bots, and training agents.
A compact reference of the AI crawlers DataEasy recognises: who runs them, what they are for, and the token they use in the User-Agent header and robots.txt.
28 crawlers found
AI answers
User-triggered / 6 crawlers
OpenAI
ChatGPT-User
Fetches a page when a ChatGPT user (or a custom GPT) asks about it or needs it to answer.
Anthropic
Claude-User
Fetches a page when a Claude user asks a question that needs it.
Perplexity
Perplexity-User
Visits pages on behalf of Perplexity users to answer their questions and cite sources.
Mistral AI
MistralAI-User
Fetches pages when Le Chat users ask questions that need fresh web content.
DuckDuckGo
DuckAssistBot
Fetches pages to generate DuckAssist answers in DuckDuckGo search results.
Meta
Meta-ExternalFetcher
Fetches individual links on behalf of users of Meta AI products, e.g. when a link is shared.
Search indexes
Discovery / 9 crawlers
OpenAI
OAI-SearchBot
Indexes websites so they can appear and be linked in ChatGPT search results.
Anthropic
Claude-SearchBot
Crawls the web to improve the quality of search results returned by Claude.
Perplexity
PerplexityBot
Indexes websites so they can be surfaced and cited in Perplexity answers.
Googlebot
Google's main crawler for Search, which also feeds AI Overviews and AI Mode.
Microsoft
bingbot
Bing's search crawler. The Bing index also powers Copilot answers and several AI search products.
Apple
Applebot
Powers search features across Apple products, such as Spotlight, Siri and Safari suggestions.
Amazon
Amazonbot
Crawls the web to improve Amazon services, such as answers given by Alexa.
DuckDuckGo
DuckDuckBot
DuckDuckGo's search crawler, used to improve its search results.
You.com
YouBot
Indexes pages for You.com's AI search and assistant.
Training crawlers
Model data / 8 crawlers
OpenAI
GPTBot
Crawls public pages that may be used to train OpenAI's generative AI models.
Anthropic
ClaudeBot
Collects public web content that may be used to train Anthropic's Claude models.
Google-Extended
A robots.txt control that lets you opt out of your content being used to train Gemini models.
Apple
Applebot-Extended
A robots.txt control to opt out of Applebot-crawled content being used to train Apple's AI models.
Meta
meta-externalagent
Crawls the web for use cases such as training AI models or improving products by indexing content.
ByteDance
Bytespider
ByteDance's web crawler, widely reported to collect data for its AI models.
Common Crawl
CCBot
Builds the free Common Crawl web archive, one of the most used datasets for training language models.
Allen Institute for AI
AI2Bot
Crawls web content used to build training datasets for open language models.
Other AI bots
AI crawler / 5 crawlers
Anthropic
anthropic-ai
Older Anthropic user-agent token that still appears in many robots.txt files.
Cohere
cohere-ai
User-agent token associated with Cohere's AI products fetching web content.
Diffbot
Diffbot
Extracts structured data from web pages for Diffbot's knowledge graph, used by AI applications.
Webz.io
omgilibot
Collects web data that Webz.io sells as datasets, including to AI companies.
Hive
ImagesiftBot
Crawls public images for reverse image search and AI image tools.
Discover when bots and AI assistants crawl your website.
DataEasy keeps bots out of your visitor stats and groups AI crawler hits into answers, indexing and training, so you can see how AI products read your website.