Diffbot
by Diffbot
Other AI bots · AI crawler
Extracts structured data from web pages for Diffbot's knowledge graph, used by AI applications.
Data-extraction and dataset crawlers whose content often ends up in AI products.
Control it with robots.txt
Well-behaved crawlers read robots.txt before crawling. To block Diffbot on your whole site:
robots.txt
User-agent: Diffbot Disallow: /
Blocking AI search and answer crawlers can also remove your pages from AI answers and citations. Weigh traffic from AI assistants before blocking them.