Bytespider
by ByteDance
Training crawlers · Model data
ByteDance's web crawler, widely reported to collect data for its AI models.
Crawlers (or robots.txt controls) that collect public content used to train AI models.
Control it with robots.txt
Well-behaved crawlers read robots.txt before crawling. To block Bytespider on your whole site:
robots.txt
User-agent: Bytespider Disallow: /
Blocking AI search and answer crawlers can also remove your pages from AI answers and citations. Weigh traffic from AI assistants before blocking them.