robots.txt mistakes that let AI crawlers through (with data)

Of the robots.txt mistakes that change what an AI crawler may read, the most common is giving a bot its own group, which makes it stop following your User-agent: * rules: in our scan of the Tranco top 20,000 on 8 November 2026, 717 of the 10,870 readable files (6.6%) lost * rules for an AI crawler this way. Others do nothing at all, such as the 99 sites that block a retired Anthropic name and leave ClaudeBot, the one in use, unblocked.

Below is each mistake with its count, what the standard and the operators say, and how five real parsers behave, including Google’s open source one. At the end: a template that passed all five, and a checklist you can run against your own file.

What are the most common robots.txt mistakes with AI crawlers?

These are the mistakes we could count across 10,870 readable robots.txt files, out of 14,671 reachable sites. The full method and the per-site data are public. A count shows what a file does, never what its owner meant, so some of these will be deliberate.

Why does naming a bot in its own group let it ignore your other rules?

Because a crawler obeys one group only. RFC 9309, the robots.txt standard, has crawlers fall back to the * group only “if no matching group exists”. Once a group names a bot, the * group stops applying to it. Google’s documentation says the same: “User agent specific groups and global groups (*) are not combined.”

So this file, which looks like it opens GPTBot up politely, also lets it into /admin/:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Allow: /

All five parsers we tested let GPTBot read /admin/x here and kept other bots out. In our scan:

• 717 sites (6.6%) have an AI bot in a group of its own that does not block ordinary pages, while the * group has Disallow rules that group lacks. The median site loses 11 * rules.
• 261 of those drop a rule covering a path like /admin, /wp-admin, /login, /account, /cart or /checkout.
• 76 sites shut everyone out with * and leave at least one named AI bot open. For some that is the point: 13 of them explicitly allow OAI-SearchBot while blocking the rest.

If you restrict an AI crawler, name it and block it. If you only want it to follow your normal rules, leave it out entirely.

Which AI bot names in robots.txt are out of date or made up?

Many of them, and they are widely copied. A rule for a name no crawler uses does nothing, and it can make you believe you are covered.

Retired names. Anthropic’s crawler page documents three bots, ClaudeBot, Claude-User and Claude-SearchBot, and as of November 2026 mentions neither anthropic-ai nor Claude-Web. In our scan 1,559 sites name one of those two and 1,285 block one by name. 99 of those 1,285 (7.7%) leave ClaudeBot unblocked. On espn.com, science.org and abcnews.com, anthropic-ai is blocked while ClaudeBot is not named at all. Google’s fetcher page gives Google-NotebookLM as the “Former agent (supported until August 2026)” of what is now Google-GeminiNotebook. 162 sites still name the old token, and only 48 of them name the new one. Meta’s crawler page does not list FacebookBot, which 790 sites name.

Names no operator documents. These are counts of sites naming each token:

Blocking those is harmless, and it does not buy what it looks like. If the traffic is real, it has to be blocked at your server or CDN.

Brand names used as tokens. 282 sites (2.6%) have a group for a bare product or company name such as ChatGPT or ChatGPT Agent (94 sites), OpenAI (67), DeepSeek (64), Grok (54), Claude (41) or Perplexity (27). All but one of those 282 also list a real token, so this is mostly noise. Results vary by parser, though: under Google’s parser and the standard’s rule a Claude group does not cover ClaudeBot, while Python’s urllib and Scrapy’s Protego treat it as a match. Write the documented token and the outcome stops depending on who reads the file.

Typos. 25 sites use near misses such as ClaudeWeb, PerplexityUser or a run-together PerplexityBotClaude-SearchBot. Another 151 add a version, like GPTBot/1.4. Google’s parser and the RFC 9309 parser from our study cut the suffix off, but Protego and urllib fail to match the bot, so drop the version.

Before copying a block list, check each token against our per-bot pages, or the operator’s own page.

Am I blocking AI search when I meant to block AI training?

Possibly, and a quarter of the sites blocking GPTBot also block the search crawler. OpenAI’s crawler page says each setting “is independent of the others” and that sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers”. So one list of AI bots pasted into one group does two jobs, and only one was probably the goal.

• 1,647 sites block GPTBot by name. 1,224 of them (74.3%) leave OAI-SearchBot unblocked, and 423 (25.7%) block it too.
• 1,509 sites block ClaudeBot by name. 371 of them (24.6%) also block Claude-SearchBot.
• Of the 2,224 sites that block at least one training crawler by name, 789 (35.5%) also name and block a search crawler.
• 584 of the 1,647 GPTBot blockers (35.5%) also block PerplexityBot, which Perplexity says “is not used to crawl content for AI foundation models”.

Whether that is an accident depends on the site, and a file cannot tell us. nytimes.com, bbc.co.uk and cnn.com block all three OpenAI bots by name, while forbes.com blocks GPTBot and explicitly allows OAI-SearchBot. One clue is the shape of the file: a single group with training and search names mixed together usually means someone pasted a list. 137 sites have GPTBot and OAI-SearchBot in the same blocking group. Which bot does what is covered in the pillar post; decide per kind, then write the groups.

Does blocking example.com also cover www and subdomains?

No. A robots.txt file applies only to the host that serves it. Google’s documentation says the rules “apply only to the host, protocol, and port number where the robots.txt file is hosted”, and Anthropic tells site owners to add its rule “for every subdomain that you wish to opt out from”. A block on www.example.com leaves docs.example.com, shop.example.com and the bare domain unaffected unless each one has its own file.

We could not count this. The scan read one host per domain, and 6,100 of the 10,870 readable files came from a www host. Listing your hosts and fetching each host’s /robots.txt is the only way to check it.

Can a blank line break a robots.txt group?

Not under the standard, and not in Google’s parser, but it can in others. The RFC 9309 grammar allows empty lines between a User-agent line and its rules. 1,226 files (11.3%) have a blank line straight after a User-agent line: 858 before a rule, and 452 before another User-agent line, which reads as a broken list (some files do both). 214 of them involve an AI bot’s name. Another 1,532 files have a blank line between two rules of one group, and 2,181 (20.1%) contain at least one of these.

Python’s urllib.robotparser treats a blank line as the end of a group, so it drops what comes after. This file blocks GPTBot in four of our five parsers and allows it in that one:

User-agent: GPTBot

Disallow: /

If a crawler audit or a compliance tool is built on urllib, it will read a blocked bot as allowed. The file is fine by the standard, and the tool is wrong, but removing the blank lines costs you nothing and ends the argument.

What happens if you have more than one User-agent: * group?

Under the standard, they are combined. RFC 9309 says that if more than one group matches, “the matching groups’ rules MUST be combined into one group”, and Google merges them too. 521 files (4.8%) have two or more * groups, and in 420 of them more than one group holds rules.

Parsers that stop at the first * group, as urllib does, see a different file. In 8 files that changes the outcome for an ordinary page, and in 14 the first * group is empty, so such a parser reads no rules at all. Keep one * group and the question goes away.

Can a copy-pasted hyphen break a User-agent line?

Yes. RFC 9309 says a product token “MUST contain only uppercase and lowercase letters”, underscores and hyphens, and a look-alike is none of those. 10 sites write the Perplexity fetcher with U+2011, the non-breaking hyphen, in place of the ordinary hyphen, and for 9 of them it is the only spelling they use.

The likely source is the documentation itself: as of 8 November 2026 the table on Perplexity’s crawler page, in both its HTML and Markdown forms, shows the bot’s name with U+2011 in the first column and the plain hyphen only in the description beside it. In all five parsers, Perplexity-User was not blocked by the U+2011 spelling. Google’s parser reads the token only up to the first character it does not allow, so it sees a group for Perplexity.

In total, 23 files have a non-ASCII character somewhere in a User-agent value. To find them, grep for anything outside ASCII, using the command in the checklist below.

Does robots.txt stop ChatGPT-User and other fetchers?

Not reliably. 776 sites (7.1%) block ChatGPT-User, Perplexity-User, Meta-ExternalFetcher or Amzn-User by name, and all four operators say those fetchers may not obey:

• OpenAI on ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.”
• Perplexity on Perplexity-User: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”
• Meta on Meta-ExternalFetcher: it “may bypass robots.txt rules”.
• Amazon on Amzn-User: “it may not follow all robots.txt directives”.

If you need them gone, block by user agent and the operator’s IP list at your server or CDN. The AI crawlers post has the lists and the wording for the rest. RFC 9309 itself says robots.txt rules “are not a form of access authorization”.

How do different parsers read these files?

We ran each case through Google’s open source robots.txt parser, compiled from source, the RFC 9309 parser from our study, Python’s urllib.robotparser, Scrapy’s Protego and the robots-parser package on npm. This tests the libraries you can run. We cannot see which code each AI crawler runs.

Which robots.txt template actually works?

Name only the crawlers you block, give them Disallow: / so losing the * rules costs nothing, and keep one * group for everyone else. This file blocks the training crawlers and leaves search and assistants following your normal rules:

# Applies only to the host that serves it. Publish it on every host you run.

# Blocked from everything, so these names lose nothing by leaving the * group.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including OAI-SearchBot, Claude-SearchBot, PerplexityBot,
# the user fetchers, Googlebot and Bingbot.
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

We ran this exact file through all five parsers for 16 user agents, on an ordinary page and on /admin/x. Every parser agreed: the eight named crawlers were blocked on both paths, and OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Googlebot, Bingbot and an unnamed bot could read the page and not /admin/x. To add or drop tokens per bot, the robots.txt generator writes the same shape and explains each line.

To check your own file against Google’s parser:

./robots robots.txt GPTBot https://example.com/page
# user-agent 'GPTBot' with URI 'https://example.com/page': DISALLOWED

robots.txt linter checklist

Each item maps to a section above. Run the two commands, then go through the list:

# Characters outside ASCII, such as a look-alike hyphen. No output means none.
grep -nP '[^\x00-\x7F]' robots.txt

# A blank line straight after a User-agent line.
awk 'tolower($0) ~ /^[ \t]*user-agent[ \t]*:/ {u=FNR} /^[ \t\r]*$/ {if (FNR==u+1) print FILENAME": blank line after User-agent, line "FNR}' robots.txt

• R1: every bot in a group of its own is either blocked or meant to skip your * rules.
• R2: no retired names without their replacement: not anthropic-ai or Claude-Web alone, and Google-GeminiNotebook next to any Google-NotebookLM.
• R3: every token appears on its operator’s page. No brand-name guesses, no version suffixes.
• R4: training and search crawlers sit in separate groups, or you have decided you want out of both.
• R5: every host you serve has its own file: bare domain, www, docs, shop.
• R6: no blank lines inside a group.
• R7: exactly one User-agent: * group.
• R8: no non-ASCII characters in a User-agent line, by the grep above.
• R9: no empty Disallow: under a bot you meant to block.
• R10: nothing relies on robots.txt to stop a fetcher that says it may ignore it.

How do you confirm the rules worked?

Fetch the file from each host after every deploy, then read your server logs after the operator’s stated delay: a compliant crawler keeps requesting /robots.txt and stops asking for the pages you closed. invisit, the tool we build, can do both parts of that. Its free AI visibility check parses your robots.txt by RFC 9309 and asks it about the AI crawlers it tracks, and @invisitdev/server records crawler reads so the Crawlers page shows which paths each bot still requests. It identifies bots by user agent, so a fake GPTBot counts as GPTBot, and neither piece stops a crawler that ignores the file.