meta-externalagent
by Meta
Training crawlers · Model data
Crawls the web for use cases such as training AI models or improving products by indexing content.
Crawlers (or robots.txt controls) that collect public content used to train AI models.
Control it with robots.txt
Well-behaved crawlers read robots.txt before crawling. To block meta-externalagent on your whole site:
robots.txt
User-agent: meta-externalagent Disallow: /
Blocking AI search and answer crawlers can also remove your pages from AI answers and citations. Weigh traffic from AI assistants before blocking them.