AI2Bot
by Allen Institute for AI
Training crawlers · Model data
Crawls web content used to build training datasets for open language models.
Crawlers (or robots.txt controls) that collect public content used to train AI models.
Control it with robots.txt
Well-behaved crawlers read robots.txt before crawling. To block AI2Bot on your whole site:
robots.txt
User-agent: AI2Bot Disallow: /
Blocking AI search and answer crawlers can also remove your pages from AI answers and citations. Weigh traffic from AI assistants before blocking them.