Googlebot

What is Googlebot?

Google's main crawler, used to find and index pages for Google Search, including Discover, Google Images, Video mode and Google News.

Crawls pages for a classic search index. Blocking it removes you from that search engine.

Google says it intends to support the AIPREF training preference signal (user-agent: __usage_ai-training) in early 2027.

Google’s documentation

Googlebot user agent

Google documents these user agents for Googlebot. Version numbers in it change over time, so match on the name, not the whole string.

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Googlebot-Image/1.0

Should you block Googlebot?

Allow it unless you want out of that search engine. Blocking a major search crawler removes your pages from its results.

How to block Googlebot in robots.txt

Add a group for its tokens to robots.txt at the root of your site.

A bot follows only the group that names it, so it ignores your User-agent: * rules once it has a group of its own. Repeat any paths you also want kept from it there.

Google-Extended is a robots.txt token, not a crawler: disallowing it stops content Google crawls from being used to train Gemini models or for grounding, without affecting Google Search.

User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Googlebot-News
Disallow: /

Build a robots.txt for AI bots

How to verify a request is really Googlebot

Anyone can send any user agent, so the string alone proves nothing.

Google publishes the IP ranges Googlebot crawls from. A request that claims to be Googlebot from any other address is not Googlebot.

Or check by reverse DNS: look up the hostname of the request’s IP, confirm it ends in googlebot.com, then look that hostname up again and confirm it points back to the same IP.

Googlebot IP ranges

See when Googlebot reads your site

Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.

Track crawlers with invisit