[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt) # ClaudeBot vs Claude-SearchBot vs Claude-User: what to block > What each Anthropic bot does, why anthropic-ai and Claude-Web are out of date, tested robots.txt files, and why an IP block can backfire. - Author: Miraç Berk - Published: 2026-11-08 - Page: https://invisit.dev/blog/claudebot-vs-claude-searchbot - Tags: crawlers, robots-txt, ai-traffic, anthropic ![A matte white mailbox with two faded frosted glass name plates and one new indigo plate, on violet square-pixel dithered ground in soft light.](https://invisit.dev/_astro/cover.B1cOm0jC.png) Block ClaudeBot to keep your future content out of Anthropic's training data, and leave Claude-SearchBot and Claude-User open if you want Claude to find and read your pages. Unlike OpenAI with ChatGPT-User, Anthropic says its user-triggered fetcher follows robots.txt, so blocking Claude-User is a real switch with a real cost. And check your file for the old names: in our scan of 10,870 readable robots.txt files on 8 November 2026, 1,285 sites still block `anthropic-ai` or `Claude-Web` by name, and 99 of them (7.7%) leave ClaudeBot unblocked. Below: what Anthropic says each bot does, what to do about the retired names, why the fetcher differs from OpenAI's, a tested robots.txt for each decision, why Anthropic wants a rule on every subdomain and warns against IP blocks, how to confirm a change, and what other sites chose. If you are deciding about OpenAI's bots too, [GPTBot vs OAI-SearchBot](/blog/gptbot-vs-oai-searchbot) covers that, and the [full list of AI crawlers](/blog/ai-crawlers) covers everyone else. ## What is the difference between ClaudeBot, Claude-SearchBot and Claude-User? ClaudeBot gathers content that could train Anthropic's models, Claude-SearchBot indexes pages for Claude's search results, and Claude-User fetches a page when a person asks Claude a question. Each has its own robots.txt token. This is [Anthropic's crawler page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), checked on 8 November 2026: | | [ClaudeBot](/bots/claudebot) | [Claude-SearchBot](/bots/claude-searchbot) | [Claude-User](/bots/claude-user) | | ----------------- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | Job | "collecting web content that could potentially contribute to their training" | "navigates the web to improve search result quality for users" | "When individuals ask questions to Claude, it may access websites" | | If you disable it | "signals that the site's future materials should be excluded from our AI model training datasets" | "prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results" | "prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search" | | robots.txt | Honoured | Honoured | Honoured | Note the word "future" in the first row. Anthropic's page is explicit that a ClaudeBot block covers materials from now on, and it says nothing about content already collected. Note also "may reduce" in the other two: Anthropic does not promise that blocking removes you from Claude's answers, only that visibility may fall. The page gives no full user agent strings, so match on the token names in your logs. It publishes one IP list for all three bots, [bots.json](https://claude.com/crawling/bots.json), which held 14 prefixes when we fetched it on 8 November 2026 and was generated on 20 October. ## Are anthropic-ai and Claude-Web still real crawlers? No. Neither name appears on Anthropic's crawler page today, and in July 2024 an Anthropic spokesperson told [404 Media](https://www.404media.co/websites-are-blocking-the-wrong-ai-scrapers-because-ai-companies-keep-making-new-ones/) that "The 'ANTHROPIC-AI' and 'CLAUDE-WEB' user agents are no longer in use." We are quoting that from the press because Anthropic's own page does not address the old names at all. The two [archived copies](https://web.archive.org/web/20240501215731/https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) of the page we could read, from May 2024 and August 2025, do not mention them either. The old names are still everywhere. In our scan, 1,472 sites name `anthropic-ai` and 1,232 name `Claude-Web`. 1,559 name at least one, and 1,285 block at least one by name. **99 of those 1,285 sites (7.7%) leave ClaudeBot unblocked.** espn.com, science.org and abcnews.com are in that group: on 8 November each blocks `anthropic-ai` and GPTBot by name and has no line for ClaudeBot. A few of the 99, such as trustpilot.com, do name ClaudeBot but only restrict some paths, so not every one is an oversight. Anthropic's position complicates that count. The same spokesperson added: "We have configured ClaudeBot, our centralized user agent, to respect any existing robots.txt directives that were previously set for these deprecated user agents." So those 99 sites may not be open in practice. But that is a 2024 statement to a reporter, Anthropic's current page does not repeat it, and under RFC 9309 a file that never names ClaudeBot gives it the `User-agent: *` rules. We tested this file: ```text title="robots.txt (stale)" User-agent: anthropic-ai User-agent: Claude-Web Disallow: / User-agent: * Disallow: /admin/ ``` Read by the standard, ClaudeBot may fetch everything except `/admin/`. The fix is one line, and the old names can stay or go: ```text title="robots.txt" User-agent: anthropic-ai User-agent: Claude-Web User-agent: ClaudeBot Disallow: / User-agent: * Disallow: /admin/ ``` If a request in your logs today says `anthropic-ai` or `Claude-Web`, be suspicious of it. By Anthropic's statement it is not sending those names, so check the IP against bots.json before you believe it. ## Does Claude-User obey robots.txt, unlike ChatGPT-User? Yes, by Anthropic's own account. Its page says "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt", and the bots it means include Claude-User. It also says disabling Claude-User "prevents our system from retrieving your content in response to a user query". OpenAI takes the opposite position. | | ChatGPT-User | Claude-User | | ------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | Operator's words | "Because these actions are initiated by a user, robots.txt rules may not apply." | Bots "respect 'do not crawl' signals"; disabling it "prevents our system from retrieving your content" | | Named as a robots.txt tag | No: OpenAI's [page](https://platform.openai.com/docs/bots) names OAI-SearchBot and GPTBot | Yes | | To stop it | At your server or CDN | robots.txt | This changes the decision. A `Claude-User` disallow has a stated effect, so it also has a stated cost: when someone asks Claude about your page, Claude cannot read it. On our own site, Claude-User made 76 requests from 4 to 8 November and 6 of them were for `/robots.txt`, which fits a fetcher that checks the file. That is one site, a few days, and possibly some of our own agents, so treat it as consistent rather than proof. One separate product does not fall under these rules. Anthropic's [hosted browser](https://support.claude.com/en/articles/18234133-how-do-anthropic-hosted-browsers-identify-themselves-when-they-browse-the-web-on-a-user-s-behalf), used by Claude Duet, sends a normal browser user agent with `Claude-Agent` added, from IPs listed at `claude.com/.well-known/sail`. Anthropic says preferences for its robots are "independent", and that page does not mention robots.txt. Control it by user agent and IP at your edge. ## What should you block for your type of site? Block ClaudeBot only if you object to training, and keep Claude-SearchBot unless you want out of Claude's search. Anthropic offers no advice by site type, so this is our judgment: | Site type | ClaudeBot | Claude-SearchBot | Claude-User | The Anthropic-specific reason | | ---------------------- | --------------------------------- | ---------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------- | | Docs or developer tool | Allow | Allow | Allow | A Claude-User block would stop Claude reading your docs when someone asks it how to use them. | | Publisher | Block | Allow | Decide per section | Claude-User is honoured, so a path rule on it can keep one section unread while the rest of the site stays readable. | | SaaS marketing site | Your call | Allow | Allow | Training affects future materials only. Search and fetches are how Claude sees your current pricing. | | E-commerce | Block if your copy sets you apart | Allow | Allow | A shopper asking Claude about a product needs Claude-User to get through. | ## What robots.txt do you need for each decision? Name only the bots you restrict. We ran each file through a matcher that follows [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), checking each Anthropic token on `/`, a normal page and a private page. If you want all three to have access, name none of them and keep only your `User-agent: *` rules. **Block training, stay in Claude's search.** ```text title="robots.txt" User-agent: ClaudeBot Disallow: / User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml ``` **Block training and search, allow user fetches.** ```text title="robots.txt" User-agent: ClaudeBot User-agent: Claude-SearchBot Disallow: / User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml ``` **Block all three.** Unlike the OpenAI version, the third name is a control Anthropic documents. ```text title="robots.txt" User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User Disallow: / User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml ``` **Block training, keep one section away from user fetches.** Claude-User's group repeats `/admin/` because a named group replaces the `*` group for that bot. Robots.txt is not access control, so anything truly private still needs a login. ```text title="robots.txt" User-agent: ClaudeBot Disallow: / User-agent: Claude-User Disallow: /members/ Disallow: /admin/ User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml ``` The [robots.txt generator](/tools/robots-txt-generator) writes files like these from choices, and its "Check my file" tab reads yours the way the RFC says. ### How do you slow ClaudeBot down instead of blocking it? Use `Crawl-delay`, which Anthropic calls a "non-standard" extension it supports, and put it in a group that also has a `Disallow` line. Anthropic does not define the unit, and its example value is 1. ```text title="robots.txt" User-agent: ClaudeBot Disallow: /admin/ Crawl-delay: 5 User-agent: * Disallow: /admin/ ``` The `Disallow` line matters. A group holding only `User-agent: ClaudeBot` and `Crawl-delay` is read two ways: our RFC 9309 matcher treated it as merged with the `*` group that follows, while Python's `urllib.robotparser` treated ClaudeBot as having no rules at all and allowed `/admin/`. Repeat your private paths inside the group and both parsers agree. ## Why does Anthropic want the rule on every subdomain? Because robots.txt only covers its own host, Anthropic asks you to add the rule "for every subdomain that you wish to opt out from". A file on `example.com` says nothing about `docs.example.com`. RFC 9309 defines the file by its authority: `scheme:[//authority]/robots.txt`. This loop prints how many `User-agent` lines name ClaudeBot on each host, and a 0 means that host has no rule for it: ```sh title="Terminal" for host in example.com www.example.com docs.example.com blog.example.com; do printf '%-22s %s\n' "$host" "$(curl -sL -m 15 "https://$host/robots.txt" | grep -ci '^[[:space:]]*user-agent:[[:space:]]*claudebot')" done ``` Pay extra attention to any subdomain that runs on a different platform from your main site, since it serves its own file. ## Why shouldn't you block Anthropic by IP address? Anthropic warns that "blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." It says opting out "requires modifying the robots.txt file". Add your rule there first. RFC 9309 shows why the distinction matters. If a robots.txt fetch gets a 4xx, the file is "unavailable" and the crawler "MAY access any resources on the server". If it fails with a server or network error, the crawler "MUST assume complete disallow". So a firewall rule can tell an RFC-following crawler two opposite things depending on how it refuses. Anthropic's page does not say how its bots treat each case. There is a practical reason too. bots.json is a single list with no per-bot split, so an IP rule cannot separate ClaudeBot from Claude-User. You would be refusing the fetcher as well as the training crawler. The list is there to confirm that a request is really Anthropic's, and [our crawler guide](/blog/ai-crawlers) has a script for that check. If you also want a server rule, match the user agent and keep `/robots.txt` reachable: ```nginx title="nginx.conf" map $http_user_agent $block_ai_fetcher { default 0; ~*ClaudeBot 1; } server { listen 80; root /usr/share/nginx/html; # Bots must still be able to read your opt-out location = /robots.txt { } location / { if ($block_ai_fetcher) { return 403; } try_files $uri $uri/ =404; } } ``` In our nginx test, ClaudeBot got 403 on pages and 200 on `/robots.txt`, and Claude-User and an ordinary browser got 200 on both. Keep the robots.txt rule even with this in place, since that is the opt-out Anthropic documents. ## Does Anthropic's new no-training signal replace the ClaudeBot rule? No. A rule naming ClaudeBot still wins. Since 30 October 2026, Anthropic's page says it recognises a machine-readable "no AI training" signal, and: "If your robots.txt file names ClaudeBot, we follow that rule, even if your general setting says something different." If the file does not mention ClaudeBot, it follows the general setting. The signal comes from [an Internet-Draft](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) by an Anthropic author, using the token `__usage_ai-training` in an ordinary group. The draft says the token "is not a directive for crawling behavior" and that wildcard groups "MUST NOT be interpreted as matching the policy token". This file says no to training everywhere and lets ClaudeBot read one section, which the matcher confirmed: ```text title="robots.txt" User-agent: __usage_ai-training Disallow: / User-agent: ClaudeBot Allow: /press/ Disallow: / User-agent: * Disallow: /admin/ ``` The draft is an individual submission, and its datatracker page says it "has no formal standing in the IETF standards process". Anthropic says it is "rolling this out across all of our services over time" and that how it reads these settings may change. For now, name ClaudeBot. ## How long do changes take, and how do you confirm them? Anthropic's page gives no timeframe. RFC 9309 says crawlers "SHOULD NOT use the cached version for more than 24 hours", so we would wait two days, then check. First confirm your server does not refuse Anthropic's bots, on the file and on a page: ```sh title="Terminal" for ua in ClaudeBot Claude-SearchBot Claude-User; do printf '%-18s robots %s page %s\n' "$ua" \ "$(curl -sL -o /dev/null -w '%{http_code}' -A "$ua" https://example.com/robots.txt)" \ "$(curl -sL -o /dev/null -w '%{http_code}' -A "$ua" https://example.com/)" done ``` Your robots.txt should answer 200 for all three. A 403 means something in front of your site is blocking the name, which is exactly the situation Anthropic warns about. This tests user agent rules only. Then read your logs. This counts requests per bot, split into robots.txt and pages with status codes, for an nginx or Apache combined log. It includes the old names, so any request still using them shows up: ```sh title="Terminal" awk -F'"' 'match($6, /ClaudeBot|Claude-SearchBot|Claude-User|anthropic-ai|Claude-Web/) { bot = substr($6, RSTART, RLENGTH) split($2, request, " "); split($3, response, " ") print bot, (request[2] == "/robots.txt" ? "robots.txt" : "pages"), response[1] }' /var/log/nginx/access.log | sort | uniq -c | sort -rn ``` A blocked bot that behaves shows `robots.txt 200` and no new `pages 200` lines. If you see pages after the waiting period, check the request's IP against bots.json first, since a user agent can be faked. If the IP is Anthropic's, its page offers a contact: claudebot@anthropic.com, written from an address at the domain in question. ## Which Anthropic bots do other sites block? Most sites that block ClaudeBot leave search open. Of the 10,870 readable robots.txt files in our scan of the Tranco top 20,000, these block each bot by name, and "effective" adds sites blocking it only through `User-agent: *`: | Bot | Sites by name | By name | Effective | | ---------------- | ------------- | ------- | --------- | | ClaudeBot | 1,509 | 13.88% | 18.08% | | Claude-User | 396 | 3.64% | 7.85% | | Claude-SearchBot | 362 | 3.33% | 7.52% | Of the 1,509 ClaudeBot blockers, 1,138 (75.4%) leave Claude-SearchBot unblocked and 371 (24.6%) block it too. Among the top 1,000 sites, 17.81% block ClaudeBot by name. For comparison, GPTBot is blocked by name on 1,647 sites. Named examples from 8 November: medium.com blocks ClaudeBot and GPTBot and leaves search open. theguardian.com blocks ClaudeBot by name and does not name GPTBot. nytimes.com blocks all three of Anthropic's bots. Firewall blocks are invisible to this method, 5,329 of the 20,000 domains did not answer, and it is one snapshot. The full method is in [how many sites block AI crawlers](/blog/how-many-sites-block-ai-crawlers). On invisit.dev, one site, from 4 to 8 November, ClaudeBot made 379 requests across 182 paths and Claude-User made 76, so training crawls outnumbered user fetches by about five to one. ## How can you see which Anthropic bots read your site? Your server logs answer this for free with the command above. A browser analytics script sees none of it, because these bots do not run it. invisit, the tool we build, records them through `@invisitdev/server` in your middleware ([crawler reads](https://invisit.dev/docs/server/sdk.md#crawler-reads)), and its Crawlers page splits AI training from AI answers and lists reads by bot, by page and by response code. It has three limits worth knowing. It classifies by user agent without checking bots.json, so a fake ClaudeBot counts as ClaudeBot. It reports `anthropic-ai` and `Claude-Web` requests under ClaudeBot rather than separately, so use the log command to see those. And it does not name `Claude-Agent`. The [AI visibility checker](/tools/ai-visibility) is a quick check of what your robots.txt tells each bot. ## Checklist before you change robots.txt - [ ] Search your file for `anthropic-ai` and `Claude-Web`, and add `ClaudeBot` next to them. - [ ] Decide each bot separately: training, search, user fetches. - [ ] Remember a Claude-User block is honoured, so it costs you Claude's answers about that page. - [ ] Publish the file on every subdomain you want covered. - [ ] Do not rely on an IP block, and keep `/robots.txt` reachable through any firewall rule. - [ ] Wait about two days, then read your logs.