How many websites block AI crawlers? Data from the top 20,000

Of the 10,870 sites in the Tranco top 20,000 with a readable robots.txt on 8 November 2026, 2,247 (20.7%) disallow at least one AI bot by name and 1,647 (15.15%) disallow GPTBot by name. CCBot is the most blocked name at 15.43%. Most sites that refuse AI training leave AI search open: 74.3% of the sites blocking GPTBot do not block OAI-SearchBot, and 75.4% of those blocking ClaudeBot do not block Claude-SearchBot.

Below are the numbers for 25 bots, the split between training, search and user-triggered fetchers, how large and smaller sites differ, the retired names sites still block, the link between blocking and Cloudflare-style signals, the full method with its limits, and the raw data, which you may reuse with attribution.

Shares are of the 10,870 sites with a readable file, except the last two, which are shares of the 1,647 GPTBot blockers and the 1,509 ClaudeBot blockers.

What share of websites block AI crawlers?

About one in five popular websites blocks at least one AI bot by name: 2,247 of 10,870 readable robots.txt files, or 20.7%. Another way to read it is that 79.5% block none of the eight main training names (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, Google-Extended, Amazonbot and Applebot-Extended) by name.

A smaller group goes all the way: 780 sites (7.2%) block all eight training names. And 2,981 (27.4%) mention at least one AI bot somewhere in the file, whether to block or to allow it.

“By name” means a robots.txt group names that bot and disallows ordinary pages. Separately, 507 sites (4.7%) shut out every bot they do not name with User-agent: * and a full disallow. Counting sites blocked only that way, GPTBot’s share rises from 15.15% to 19.15%. We report both, and use the by-name figure wherever we talk about a decision aimed at a specific bot.

Which AI crawlers get blocked the most?

CCBot, GPTBot and Bytespider are the most blocked names, each on about 15% of readable files. Training crawlers are blocked far more often than search crawlers or user fetchers: most named training bots sit between 11.75% and 15.43%, the search bots between 0.70% and 6.40%, and the user-triggered fetchers between 1.67% and 5.83%.

The Kind column is our grouping; what each bot does is in our AI crawler reference. Two placements are judgement calls. Common Crawl describes CCBot as building “an open repository of web crawl data”, and ByteDance’s only page on Bytespider (in Chinese) describes it as Toutiao Search’s crawler. They sit with training here because their block rates do. Google-Extended is also not a crawler: Google calls it “a standalone product token”, and blocking it does not touch Google Search (Google, checked on 8 November 2026).

Almost nobody blocks the search engines themselves. Googlebot is disallowed by name on 20 sites of 10,870.

Do sites that block AI training also block AI search?

Mostly no. Of the 1,647 sites that block GPTBot by name, 1,224 (74.3%) leave OAI-SearchBot unblocked and 423 (25.7%) block it too (412 by name, 11 through a User-agent: * rule). The pattern is almost identical at Anthropic: of the 1,509 sites blocking ClaudeBot by name, 1,138 (75.4%) leave Claude-SearchBot unblocked and 371 (24.6%) block it too (359 by name, 12 through *). This is the study’s main finding. A site refusing training usually stays available for citation.

Part of this split may be a default rather than a decision. 519 of the 1,224 GPTBot blockers that leave search open also carry a Content-Signal line, the mark of a managed robots.txt. Without those sites the pattern weakens but holds: of the 1,101 GPTBot blockers with no Content-Signal line, 705 (64.0%) leave OAI-SearchBot unblocked, and of the 969 such ClaudeBot blockers, 625 (64.5%) leave Claude-SearchBot unblocked.

The opposite setup barely exists. Only 2 sites block OAI-SearchBot by name while leaving GPTBot open. 13 sites shut out every crawler with * but name OAI-SearchBot with an explicit allow, so they let exactly the search side in.

The split is one operators invite. OpenAI says “each setting is independent of the others” and that a site “can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot” (OpenAI, checked on 8 November 2026). It also says sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links”. Anthropic says disabling Claude-SearchBot “may reduce your site’s visibility and accuracy in user search results” (Anthropic, checked on 8 November 2026).

Part of this split may be a default rather than a decision. The managed file Cloudflare documents names training crawlers only, so a site using it gets the selective setup without writing a line (more below).

Do websites block user-triggered fetchers like ChatGPT-User?

Far less often than training bots, and the block may do less than it looks like. ChatGPT-User is disallowed by name on 634 sites (5.83%), against 1,647 for GPTBot. Of the GPTBot blockers, 1,032 (62.7%) leave ChatGPT-User unblocked, and 381 block all three OpenAI bots by name.

A user fetcher loads a page because someone asked an assistant about it, so operators treat robots.txt differently here, and they do not agree with each other. As of 8 November 2026:

So the 634 sites disallowing ChatGPT-User have recorded a preference that OpenAI says may not apply. Where a fetcher ignores robots.txt, only a server or CDN rule actually stops it.

Do bigger websites block AI crawlers more than smaller ones?

Yes, on every bot we split by rank, and in relative terms the gap is widest for the search and user bots. GPTBot is blocked by name on 18.00% of readable files in the top 1,000 and 15.15% in the top 20,000. OAI-SearchBot nearly doubles, from 3.83% across the top 20,000 to 7.24% in the top 1,000.

The top tier is a small base: 511 readable files in the top 1,000, against 5,472 in the top 10,000 and 10,870 overall. The tiers are nested, so the top 20,000 column contains the other two.

Which retired or undocumented bot names are sites still blocking?

Well over a thousand robots.txt files name bots their operators do not document. 1,559 sites name anthropic-ai or Claude-Web, and 1,367 name cohere-ai. Of the 1,285 sites that block one of the two old Anthropic names by name, 99 (7.7%) leave ClaudeBot, the name Anthropic uses today, unblocked. Examples include espn.com, science.org and abcnews.com, which block anthropic-ai and never mention ClaudeBot.

Anthropic. Its crawler page does not mention anthropic-ai or Claude-Web at all, and says nothing about their status. The retirement comes from a statement an Anthropic spokesperson gave 404 Media in July 2024: the two names “are no longer in use”, and ClaudeBot was configured “to respect any existing robots.txt directives that were previously set for these deprecated user agents”.

That qualifies the 99. Read literally under RFC 9309, these files leave ClaudeBot unblocked. Anthropic’s 2024 statement says the old rules were carried over, but that is a press statement from two years ago, and its current page does not repeat it. A few of the 99, such as trustpilot.com and oreilly.com, do name ClaudeBot with rules of their own that leave ordinary pages open. If you mean to refuse Anthropic’s training crawler, name ClaudeBot, which is the rule Anthropic’s page describes.

Cohere. Its crawler page says “We do not use Cohere bots or user agents for the purpose of crawling or scraping web content to train generative AI foundation models at this time”, and the table of bots reads N/A. The string cohere-ai is not given as a bot name; the page’s only example token is Coherebot, in instructions for a future bot. The statement covers training crawlers and is silent on other kinds of bot.

xAI. On 8 November 2026 we downloaded xAI’s documentation export (about 2 MB) and searched it. “robots” and “crawler” appear zero times, the only grokbot matches are an email domain and a download path, and the only xai-grok matches are a package called xai-grok-shell. The names in these files come from somewhere other than xAI’s documentation.

Google. Google’s fetcher page gives Google-NotebookLM as a former agent, supported until August 2026.

The CSV also tracks FacebookBot (named on 790 sites) and DeepSeekBot (named on 300). We could not confirm the status of either from the operator’s own pages on the day, so we make no claim about them.

Is Cloudflare behind the blocking?

We can’t say from robots.txt alone, but the correlation is strong. 762 sites (7.0%) carry a Content-Signal line, the directive from Cloudflare’s Content Signals, and 546 of those (71.7%) block GPTBot by name, against 1,101 of the other 10,108 (10.9%). Those 546 sites are 33.2% of all GPTBot blockers. The most common value, on 466 sites, is search=yes,ai-train=no,use=reference.

Cloudflare’s docs, last updated 2 November 2026, explain why the two might travel together. When a customer turns on Bot Preference Sync and sets a policy to block, Cloudflare prepends managed content to the site’s robots.txt. The example file on that page has a Content-signal line with those same three values and a Disallow: / for eight names: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. Those are the same eight names as our main training list, and 780 sites block all eight.

This is a correlation. It does not show that any particular site uses Cloudflare:

• Cloudflare publishes a generator for these lines, so anyone can write one by hand or copy one.
• We did not check which sites sit behind Cloudflare, and a robots.txt does not say.
• Cloudflare documents a separate case: Free plan domains with no robots.txt of their own may display the Content Signals Policy, which “does not express any specific preferences”. That text is all comments, and our parser ignores comments, so those files are not among the 762.

Two consequences matter for reading the rest of this study. The example lists only training crawlers, so a site using it would block GPTBot and leave OAI-SearchBot alone, which is the selective pattern reported above. The content_signal column in the CSV lets you separate those sites. And Cloudflare says setting Training to Disallow also “enables edge blocking for AI training crawlers”, so for those sites the block may be enforced, which robots.txt cannot show.

The newer shared training signal has barely started: __usage_ai-training appears on 22 sites (0.2%). Google says it “intends to support” it “in early 2027” (Google). Anthropic’s page says that as of 30 October 2026 it has begun recognising a “no AI training” signal based on IETF aipref work, and that a robots.txt rule naming ClaudeBot takes priority over it, without giving the token in the page’s text.

How many websites have an llms.txt file?

2,053 of the 14,671 reachable sites (14.0%) serve an llms.txt, rising to 130 of 699 (18.6%) in the top 1,000. Of the 2,053, 1,902 open with a # heading, the one section the format requires. We checked the site root only, and the spec also allows files at subpaths, so this undercounts. The details are in our llms.txt adoption post.

What should you check in your own robots.txt?

Run this against your own domain to see which Anthropic, Cohere and xAI names your file uses, then compare with the checklist:

curl -s https://www.example.com/robots.txt \
  | grep -iE '^[[:space:]]*user-agent:[[:space:]]*(anthropic-ai|claude-web|cohere-ai|grokbot|claudebot|claude-searchbot|claude-user)[[:space:]]*$'

• If you see anthropic-ai or Claude-Web without ClaudeBot, add a ClaudeBot rule. Do not rely on a 2024 statement.
• Decide training and search separately. Blocking GPTBot leaves OAI-SearchBot alone, and that may be exactly what you want.
• Blocking a user fetcher is a preference some operators say they may not apply. Use a server or CDN rule if it must hold.
• If your CDN manages robots.txt, read the file it actually serves. The rules you wrote may not be all of it.
• Publish the file on every subdomain you care about; Anthropic asks for this explicitly.

The robots.txt generator writes a current file, and the AI crawler reference has the full list of tokens.

How was this measured?

We fetched robots.txt for every domain in the Tranco top 20,000, list 94P72, generated on 7 November 2026, and parsed each file the way RFC 9309 specifies.

• Fetching. 8 November 2026, 14:56 to 15:09 UTC. We requested https://<domain>/robots.txt, then the www host, plus /llms.txt on the same origin. The user agent was invisit-research/1.0 (+https://invisit.dev/bots; help@invisit.dev).
• Readable file. HTTP 200, not HTML, and containing directives or empty. 14,671 of the 20,000 domains answered and 10,870 served a readable file (511 in the top 1,000, 5,472 in the top 10,000). 186 files were empty.
• Parsing. All groups naming a token are merged, * applies only when no group names the token, the longest match wins, and Allow wins a tie, per RFC 9309.
• Blocked. An ordinary page path (we probe /invisit-probe/some-article) is disallowed for that token.
• By name. A group names the token and disallows ordinary pages. “Including User-agent: *” adds sites that block the token only through the wildcard.
• “Leaves it unblocked”. In the training versus search comparison, the training bot must be blocked by name, and the search bot counts as blocked whether by name or through *.

Two checks on the pipeline. Our parser agreed with Python’s urllib.robotparser on 4,473 of 4,500 comparisons (1,500 wildcard-free files, 3 bots). All 27 differences were urllib departing from the RFC: 18 from dropping rules after a blank line, and 9 from reading only the first of several User-agent: * groups. And to test whether sites serve bots a different file, we refetched 302 sites with a Chrome user agent; of the 287 that answered, 286 served the same file.

The limits are real and worth stating before you quote any of this:

• Tranco ranks domains, not websites. The list includes CDNs, ad servers and API hosts with no site behind them.
• Percentages are of readable files. 5,329 of the 20,000 did not answer (4,425 connection errors, 904 timeouts), and 1,386 answered 403. Domains with no readable file are excluded rather than counted as not blocking, so a share of all 20,000 would be lower.
• One snapshot. A robots.txt can change the next day, and we have no earlier scan to compare with.
• Partial rules are not blocks. A site disallowing GPTBot from /articles/ only counts as not blocking it, because an ordinary page path stays open.
• The apex and www only. Rules on other subdomains are not counted.
• Intent, not enforcement. RFC 9309 says these rules “are not a form of access authorization”. Firewall and CDN blocks are invisible here, and nothing in this study shows whether any bot obeyed.

Can I reuse this data?

Yes. The data may be reused with attribution. Everything is in one folder; please credit Tranco for the ranking as well as this study for the scan.

Each bot column holds one of named_block, named_partial, named_allow, star (blocked only by *) or none, and is blank where the site had no readable file. The CSV holds these derived states, not copies of each site’s file. To reproduce the main finding:

import csv

rows = list(csv.DictReader(open("sites.csv", encoding="utf8")))
valid = [r for r in rows if r["robots_valid"] == "1"]
gptbot = [r for r in valid if r["GPTBot"] == "named_block"]
search_open = [r for r in gptbot if r["OAI-SearchBot"] not in ("named_block", "star")]
print(len(valid), len(gptbot), len(search_open), round(100 * len(search_open) / len(gptbot), 1))
# 10870 1647 1224 74.3

To cite the study:

invisit (2026). How many websites block AI crawlers? Data from the top 20,000.
Scan of 8 November 2026, Tranco list 94P72.
https://invisit.dev/blog/how-many-sites-block-ai-crawlers

The ranking comes from Tranco, list 94P72. Tranco asks users to cite its paper: Le Pochat, Van Goethem, Tajalizadehkhoob, Korczyński and Joosen, “Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation”, NDSS 2019.

robots.txt only shows what a site asks for; to see which crawlers actually read your pages you need server-side records, which invisit, the tool we build, keeps with @invisitdev/server, classifying by user agent without checking IP lists.