Does blocking AI crawlers hurt SEO? What each block costs

Blocking AI training crawlers does not hurt your Google rankings: Google says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”, and GPTBot and ClaudeBot are other companies’ crawlers, absent from Google’s crawler list. Blocking AI search crawlers has a real cost, in that AI product only: OpenAI says sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers”. Blocking Googlebot removes you from Google itself. In our study, 1,647 of 10,870 sites block GPTBot by name and 74.3% of them leave OAI-SearchBot open.

Below is a direct answer for each bot family, in each operator’s own words as of 8 November 2026, then a cost table, what the top 20,000 sites actually chose, and what nobody has measured. To put any of this in place, see how to block AI bots.

Does blocking Google-Extended hurt my Google rankings?

No, by Google’s own statement. Google’s crawler documentation, checked on 8 November 2026, says: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”

It does have a cost outside Search. Google-Extended decides whether Google may use your content for “training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding”, where grounding means giving the model Google’s index at the moment of a prompt. Blocking it can therefore keep your pages out of answers in the Gemini app. It does not touch AI Overviews or AI Mode, which have their own controls (below). Google’s help page does say that Google-Extended limits training of “the models used to generate responses in Search generative AI features”, so it is the training control there too.

Google-Extended is a robots.txt token. It never appears in your logs, because Google crawls with its existing user agents.

Does blocking GPTBot or ClaudeBot affect Google?

No. Neither company has written that sentence, because nothing connects them: GPTBot is OpenAI’s crawler, and Google’s documentation describes only its own crawlers’ effect on Search. A robots.txt group naming GPTBot applies only to GPTBot, and Google says it is “crawling preferences addressed to the Googlebot user agent” that affect Google Search.

Within OpenAI’s own products, GPTBot and search are separate too. OpenAI says “Each setting is independent of the others”, and that a site can allow OAI-SearchBot “while disallowing GPTBot to indicate that crawled content should not be used for training”. Anthropic has the same split: blocking ClaudeBot “signals that the site’s future materials should be excluded from our AI model training datasets” (Anthropic, checked 8 November 2026), and it states no effect on search.

Neither operator says a training block has no effect on how often its assistant mentions you. Whether a model that was not trained on your pages names your brand less often is the open question, and we come back to it below.

Does blocking OAI-SearchBot remove me from ChatGPT?

Yes, from ChatGPT’s search answers. OpenAI, checked on 8 November 2026: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” Changes take about 24 hours.

ChatGPT-User is a separate decision. OpenAI says it is “not used to determine whether content may appear in Search” and that “robots.txt rules may not apply” to it because a person started the fetch. Blocking it at a firewall stops ChatGPT from opening your page when someone asks, without changing whether you are eligible for search answers.

The same logic holds for the other assistants, in their operators’ words:

• Claude. Disabling Claude-SearchBot “may reduce your site’s visibility and accuracy in user search results”, and disabling Claude-User “may reduce your site’s visibility for user-directed web search” (Anthropic).
• Perplexity. “To ensure your site appears in search results, we recommend allowing PerplexityBot” (Perplexity).
• Meta AI. “Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI’s responses” (Meta).
• DuckDuckGo. Blocking DuckAssistBot opts you out of its AI answers and “does not impact organic search rankings and will not affect whether or not websites appear in our search results” (DuckDuckGo).

How do I stay out of AI Overviews without leaving Google?

Use the Search generative AI control in Search Console. Since 31 August 2026 this setting has been available “to all websites worldwide”, and it targets Google’s AI features on their own. Google’s help page lists AI Overviews, AI Mode and generative AI features in Discover. Excluding your site means “You won’t receive any traffic or impressions from these features”, and the control “isn’t used as a ranking or inclusion signal affecting other parts of Search”. Find it under Settings, then Search generative AI. Google says exclusion generally takes a few days.

Before this control existed, Googlebot was the crawl control, and Google’s AI features page (last updated 10 December 2025, and it does not mention the new setting) still says “robots.txt directives for Googlebot is the control”. Disallowing Googlebot does keep you out of AI Overviews, because Google says a page must be “indexed and eligible to be shown in Google Search with a snippet”. It also takes you out of Search crawling altogether, which is the expensive way to get a small result.

nosnippet is a blunt tool: Google says it “applies to all forms of search results” and “will also prevent the content from being used as a direct input for AI Overviews and AI Mode”. You lose the snippet in ordinary results as well. Google also warns that a page disallowed in robots.txt “can still be indexed if linked to from other sites”, and a noindex tag can only be read if crawlers are allowed to fetch the page, so do not combine the two.

The search generative control has one caveat of its own. Google says your content “may still be used to power Google Search as a whole, like helping our systems better understand the language of users’ queries”. Excluding your site stops it from appearing in those features, and Google does not claim it stops all use.

What does each AI crawler block cost?

The table gives each operator’s stated consequence, as of 8 November 2026. “Not stated” means its page does not say.

Two notes on Microsoft and Apple, which do not split their bots the way OpenAI does. Bing has no robots.txt switch for Copilot grounding: today the noarchive tag keeps a page out of AI answers, and from 1 February 2027 grounding becomes a page-level setting, as the AI crawler reference explains. On that change Microsoft adds a sentence worth keeping: “Neither choice guarantees inclusion, traffic, or payment.” Apple’s opt-out for AI answers in Siri and Search is a nosnippet tag; Applebot-Extended covers training only.

What did sites actually choose?

Most sites block nothing, and most of those that block training leave search open. These figures come from our 8 November 2026 study of the Tranco top 20,000, with percentages taken over the 10,870 sites with a readable robots.txt. “By name” means a group names the bot and disallows ordinary pages.

The training-versus-search pattern is the clearest signal in the data:

• Of the 1,647 sites blocking GPTBot by name, 1,224 (74.3%) leave OAI-SearchBot unblocked.
• Of the 1,509 blocking ClaudeBot by name, 1,138 (75.4%) leave Claude-SearchBot unblocked.
• The backwards setup, blocking OAI-SearchBot while leaving GPTBot open, appears on 2 sites.
• 13 sites shut out everyone with * but name OAI-SearchBot with an explicit allow.

Named examples, on 8 November 2026: nytimes.com, bbc.co.uk and cnn.com block all three OpenAI bots by name. forbes.com blocks GPTBot and explicitly allows OAI-SearchBot. medium.com blocks GPTBot and ClaudeBot and leaves search open. wikipedia.org, shopify.com and wordpress.com name none of the three OpenAI bots or ClaudeBot.

The top of the list is a bit stricter. By name, 18.00% of the top 1,000 block GPTBot, against 15.15% of the top 20,000. These numbers show what sites chose and say nothing about what it did for them. The study covers robots.txt only, so firewall blocks do not appear in it, and Tranco also ranks infrastructure domains that are not websites.

What has nobody measured?

No operator publishes what a block costs in traffic. OpenAI, Anthropic, Perplexity, Google and Microsoft each tell you what a control does, and none of the pages we read gives a figure for lost or kept visits, citations or revenue. Google comes closest: it offers a Generative AI performance report that “can help you get an idea of how changing your control may impact traffic to your site”, for your own site only.

We searched on 8 November 2026 and found one independent causal estimate. It is not an operator’s statement, so read it as separate evidence. Zhao and Berman’s working paper, revised in April 2026, compares news publishers before and after they added a robots.txt block, and reports “approximately a 7% decline in traffic within 6 weeks”. Its limits matter:

• The sample is news publishers, up to the top 500. For lower-ranked publishers the estimate “is no longer negative”, and over longer windows the effect becomes statistically insignificant after about 20 weeks.
• Only one of three data sources has an interval that excludes zero: SimilarWeb at -0.074 [-0.141, -0.007]. Semrush and the human-browsing Comscore panel are imprecise.
• It counts any GenAI crawler block as treatment, so it cannot separate blocking GPTBot from blocking OAI-SearchBot, which is the decision this post is about.
• The data stops in May 2024, before AI Overviews, and the authors note it does “not capture consumption that takes place entirely within AI interfaces”.

So these questions have no public answer we could find:

• Does a training-only block change how often a model names your brand? This cannot be observed from outside the model.
• What does blocking OAI-SearchBot or PerplexityBot cost a small site in clicks? Nobody has published this split.
• What happens to classic Google clicks after a site uses the Search generative AI control? It has only been global since 31 August 2026, and we found no public results.
• Does the effect differ outside news, for example for SaaS or ecommerce pages?

Our own data is one small site and proves little. In a 30-day window ending 8 November, invisit.dev logged 127 live assistant fetches and 0 visits with an AI referrer, while recording only 4 Google clicks. Some of those fetches may be our own agents, and crawler recording began on 4 November. It shows that an assistant can read a page many times without a visible click, and nothing about what a block would change.

So should I block AI crawlers?

Decide per kind, because the costs differ. Treat this as a judgment call, since the evidence above stops short of a measured answer.

To see the effect on your own site, record your baseline first. Search Console’s Generative AI performance report covers Google, and your referrer data covers clicks from assistants. invisit, the tool we build, shows crawler reads on its Crawlers page and visits referred by AI assistants in the AI tab of its Sources card, which tells you whether a block changed who reads you and who arrives. It cannot see what happens inside an assistant, and Google’s AI clicks arrive inside your ordinary Google traffic, since Google counts them under the “Web” search type.

• Write down why you are blocking: training, server load, licensing or something else.
• Block by kind, and never list Googlebot, bingbot or Applebot to avoid AI answers.
• Keep a dated note of the change next to your traffic data.
• Re-read your operators’ pages every few months. Google’s control and Microsoft’s preferences are both new in 2026.