Google-Agent

What is Google-Agent?

Used by agents hosted on Google infrastructure to navigate the web and take actions when a user asks.

Fetches a page because a person asked an assistant about it, right now. Operators often say robots.txt does not apply to these user-triggered fetches.

Google says user-triggered fetchers generally ignore robots.txt because a user asked for the fetch.

Google’s documentation

Google-Agent user agent

Google documents these user agents for Google-Agent. Version numbers in it change over time, so match on the name, not the whole string.

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36

Should you block Google-Agent?

Usually allow it. Each fetch is a person asking about your page right now, which is close to a visit. Blocking it in robots.txt may not stop it, since operators often treat these fetches as the person’s own.

How to block Google-Agent in robots.txt

Google says Google-Agent does not follow robots.txt, so a robots.txt rule won’t stop it. Block it at your server or firewall by user agent, and by IP if Google publishes its ranges.

Google is experimenting with signing Google-Agent requests using Web Bot Auth, with the https://agent.bot.goog identity.

Build a robots.txt for AI bots

How to verify a request is really Google-Agent

Anyone can send any user agent, so the string alone proves nothing.

Google publishes the IP ranges Google-Agent crawls from. A request that claims to be Google-Agent from any other address is not Google-Agent.

Google-Agent IP ranges

See when Google-Agent reads your site

Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.

Track crawlers with invisit