[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt)

# Googlebot

> Googlebot is a search engine crawler run by Google. Google's main crawler, used to find and index pages for Google Search, including Discover, Google Images, Video mode and Google News.

- Operator: Google
- Kind: Search engine crawler
- robots.txt token: Googlebot, Googlebot-Image, Googlebot-Video, Googlebot-News
- Follows robots.txt: yes
- Source: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers (checked 2026-11-05)
- Page: https://invisit.dev/bots/googlebot

## What is Googlebot?

Google's main crawler, used to find and index pages for Google Search, including Discover, Google Images, Video mode and Google News.

Crawls pages for a classic search index. Blocking it removes you from that search engine.

Google says it intends to support the AIPREF training preference signal (user-agent: __usage_ai-training) in early 2027.

- [Google’s documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

## Googlebot user agent

Google documents these user agents for Googlebot. Version numbers in it change over time, so match on the name, not the whole string.

```
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Googlebot-Image/1.0
```

## Should you block Googlebot?

Allow it unless you want out of that search engine. Blocking a major search crawler removes your pages from its results.

## How to block Googlebot in robots.txt

Add a group for its tokens to robots.txt at the root of your site.

A bot follows only the group that names it, so it ignores your `User-agent: *` rules once it has a group of its own. Repeat any paths you also want kept from it there.

Google-Extended is a robots.txt token, not a crawler: disallowing it stops content Google crawls from being used to train Gemini models or for grounding, without affecting Google Search.

```
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Googlebot-News
Disallow: /
```

- [Build a robots.txt for AI bots](https://invisit.dev/tools/robots-txt-generator)

## How to verify a request is really Googlebot

Anyone can send any user agent, so the string alone proves nothing.

Google publishes the IP ranges Googlebot crawls from. A request that claims to be Googlebot from any other address is not Googlebot.

Or check by reverse DNS: look up the hostname of the request’s IP, confirm it ends in `googlebot.com`, then look that hostname up again and confirm it points back to the same IP.

- [Googlebot IP ranges](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)

## See when Googlebot reads your site

Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.

- [Track crawlers with invisit](https://invisit.dev/docs/server/sdk#crawler-reads)

## Common questions

### What is Googlebot?

Googlebot is a search engine crawler run by Google. Google's main crawler, used to find and index pages for Google Search, including Discover, Google Images, Video mode and Google News.

### Is Googlebot safe to block?

Allow it unless you want out of that search engine. Blocking a major search crawler removes your pages from its results.

### How do I block Googlebot?

Add “User-agent: Googlebot” followed by “Disallow: /” to your robots.txt.