Googlebot
by Google
Search indexes · Discovery
Google's main crawler for Search, which also feeds AI Overviews and AI Mode.
Blocking Googlebot removes you from Google Search. Use Google-Extended to opt out of Gemini training instead.
Crawlers that build the search indexes used by search engines and AI search features.
Control it with robots.txt
Well-behaved crawlers read robots.txt before crawling. To block Googlebot on your whole site:
robots.txt
User-agent: Googlebot Disallow: /
Blocking AI search and answer crawlers can also remove your pages from AI answers and citations. Weigh traffic from AI assistants before blocking them. Official documentation: developers.google.com.