CCBot
by Common Crawl
Training crawlers · Model data
Builds the free Common Crawl web archive, one of the most used datasets for training language models.
Crawlers (or robots.txt controls) that collect public content used to train AI models.
Control it with robots.txt
Well-behaved crawlers read robots.txt before crawling. To block CCBot on your whole site:
robots.txt
User-agent: CCBot Disallow: /
Blocking AI search and answer crawlers can also remove your pages from AI answers and citations. Weigh traffic from AI assistants before blocking them. Official documentation: commoncrawl.org.