AI crawler user agents in 2026: what each bot does, how to block it

AI companies read your site with three kinds of bot: training crawlers such as GPTBot and ClaudeBot, search crawlers such as OAI-SearchBot and PerplexityBot that decide whether an assistant can cite you, and fetchers such as ChatGPT-User that load a page because a person asked about it just now. Most operators give each kind its own robots.txt token, so you can refuse training and still appear in AI answers.

Below is every AI crawler, fetcher and opt-out token documented by OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, ByteDance, Common Crawl, Mistral, DuckDuckGo and Microsoft, taken from each operator’s own page and checked on 3 November 2026. After the table: a robots.txt that blocks training without leaving AI search, how to catch a fake bot, and how to see which ones read you.

What are the three kinds of AI crawler?

An AI crawler does one of three jobs: it collects pages to train models, builds the index an assistant searches and cites, or fetches one page live because someone asked. Most operators now send a separate user agent for each job, which is what makes selective blocking possible.

Only the search and user-fetch kinds put you in front of people. OpenAI says sites that block OAI-SearchBot “will not be shown in ChatGPT search answers”, though they can still appear as navigational links, and Anthropic says blocking Claude-SearchBot “may reduce your site’s visibility” in Claude’s search results. OpenAI adds that each of its robots.txt settings “is independent of the others”, so blocking GPTBot leaves OAI-SearchBot alone.

Google, Apple and Microsoft do not split their bots this way. Googlebot, Applebot and bingbot crawl for search and for the AI features built on it, such as Google’s AI Overviews and AI Mode. Training is controlled separately: with a robots.txt token at Google and Apple, with page tags at Microsoft. Blocking one of these search crawlers to stay out of AI answers removes you from that search engine too.

Every AI crawler user agent, by operator

This is every AI-related crawler, fetcher and robots.txt token the operators document, as of 3 November 2026. The robots.txt column reports what the operator itself says: quoted phrases are its words, and “not stated” means its page is silent.

Last checked on 3 November 2026 against every page linked above. We re-check this table whenever an operator adds, renames or retires a bot, and update the date here.

Three entries are easy to miss. OpenAI documents OAI-AdsBot, which only visits landing pages submitted as ChatGPT ads. Mistral now documents three bots instead of one. And Google’s NotebookLM fetcher is now Google-GeminiNotebook; Google supported the old Google-NotebookLM name until August 2026.

Operators that document no crawler

xAI publishes no crawler page. On 3 November 2026 we downloaded xAI’s full documentation export, docs.x.ai/llms-full.txt (about 2 MB), and searched it: “robots” and “crawler” each appear zero times, and the only GrokBot matches are an email domain and an app download path. Any Grok user agent you find in a bot list is not from xAI’s documentation.

Cohere’s crawler page says Cohere does not use bots to crawl or scrape web content for training “at this time”, and its table of bots reads N/A. The cohere-ai strings that circulate in bot lists are not on it.

How do I block AI training but stay in AI search?

Put the training crawlers and the training tokens in one robots.txt group with Disallow: /, and leave the search crawlers and user fetchers out of it so they follow your User-agent: * rules. This file does that for every operator in the table:

# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Shared AI training preference: Microsoft reads it from 1 February 2027,
# Google says early 2027
User-agent: __usage_ai-training
Disallow: /

# Everyone else, including OAI-SearchBot, Claude-SearchBot, PerplexityBot,
# Googlebot, Applebot, bingbot and the user fetchers
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

A group with several User-agent lines applies its rules to each of them, as RFC 9309, the robots.txt standard, defines. The caveats each operator states:

• Google-Extended does not touch Google Search. Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”. It covers Gemini training and grounding in Gemini Apps and Vertex AI. AI Overviews and AI Mode are part of Search, and Google’s AI features page names Googlebot’s robots.txt rules as the control there, with nosnippet, data-nosnippet, max-snippet and noindex to limit what is shown in all of Search.
• Google will read the shared training line. Google’s crawler page says it “intends to support” user-agent: __usage_ai-training “in early 2027”. Until then, Google-Extended is the switch.
• Applebot-Extended covers training only. Apple also uses Applebot data to answer “broad world knowledge questions” in Siri and Search, and the opt-out there is a nosnippet meta tag. If your robots.txt names Googlebot but not Applebot, Applebot follows the Googlebot rules.
• Bing has no robots.txt switch for Copilot. Today the noarchive robots meta tag keeps a page out of Copilot answers and out of Microsoft’s model training, and nocache limits both to the URL, title and snippet (Bing, 2023). From 1 February 2027 Microsoft reads __usage_ai-training in robots.txt for training, while grounding in Copilot stays a page-level ai-grounding setting; after 1 March 2027 it stops reading noarchive and nocache for AI (Bing, 30 October 2026).
• Amazon’s bots are independent. Blocking Amazonbot leaves Amzn-SearchBot alone, and Amazon also reads a noarchive meta tag as “do not use the page for model training”.
• Bytespider is left out on purpose. ByteDance’s only page describes it as Toutiao Search’s crawler and does not mention robots.txt, so a rule there has no stated commitment behind it. Block Bytespider at your server or CDN if you want it gone.
• Anthropic wants the rule on every subdomain. It also warns that blocking its IP addresses instead “may not work correctly”, because ClaudeBot then cannot read your robots.txt.
• Changes take time. OpenAI quotes about 24 hours for search, Perplexity up to 24 hours, Meta caches robots.txt for up to 24 hours, DuckDuckGo takes 72 hours, and Amazon may use a copy up to 30 days old.

Do ChatGPT-User and other user-triggered fetchers obey robots.txt?

Mostly not. OpenAI, Perplexity, Google, Meta and Amazon all say their user-triggered fetchers may ignore robots.txt because a person asked for the page. Anthropic, Mistral and DuckDuckGo say their fetchers follow it.

Blocking a user fetcher mostly costs you reach: each fetch means someone asked an assistant about your page. Where a fetcher ignores robots.txt, the only real block is at your server or CDN, matching the user agent and the operator’s IP list. RFC 9309 says outright that robots.txt rules “are not a form of access authorization”.

How do I tell a real AI crawler from a fake one?

Check the request’s IP address against the list the operator publishes, or, where the operator documents it, run a reverse DNS lookup and confirm the name with a forward lookup. The user agent alone proves nothing: anyone can send GPTBot in a header, and Common Crawl says it is “aware of crawlers falsely identifying themselves as CCBot”.

Five operators document reverse DNS:

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Both lookups must agree: the name ends in the operator’s domain, and it resolves back to the same address. A reverse record on its own is set by whoever holds the IP block.

OpenAI, Anthropic, Perplexity, Amazon, Mistral and DuckDuckGo publish IP lists only, and Meta’s crawler page publishes neither. Every JSON list in the table has the same shape, a prefixes array of ipv4Prefix or ipv6Prefix entries, so one script checks them all:

import ipaddress, json, sys, urllib.request

url, ip = sys.argv[1], ipaddress.ip_address(sys.argv[2])
request = urllib.request.Request(url, headers={"User-Agent": "ip-check"})
prefixes = json.load(urllib.request.urlopen(request))["prefixes"]
networks = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
print("listed" if any(ip in net for net in networks) else "not listed")

python3 check_ip.py https://openai.com/gptbot.json 132.196.86.10
# listed

Fetch the lists on a schedule instead of pasting addresses into a config once. When we checked on 3 November 2026, OpenAI’s chatgpt-user.json held 235 prefixes and had been regenerated on 27 October, both Perplexity lists on 28 October, and Google’s crawler lists on 2 November. Bing tells you to refresh its list daily.

Signed requests are the newer route. OpenAI signs requests from ChatGPT Work’s Cloud browser with HTTP Message Signatures (RFC 9421), and each carries a Signature-Agent: "https://chatgpt.com" header you verify against keys OpenAI publishes (OpenAI). Google says it is experimenting with the same Web Bot Auth protocol for Google-Agent, under the identity https://agent.bot.goog.

How can I see which AI crawlers visit my site?

Your server or CDN access logs record every crawler request with its user agent. Browser analytics cannot: Google Analytics 4 excludes known bots automatically, and Google says “you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded” (GA4 help).

Script-based tools also depend on the crawler running JavaScript. Of the operators above, only Google, Apple and Microsoft say their crawlers render pages (Googlebot in a headless Chromium, bingbot with Microsoft Edge’s engine), and GA4 excludes known bots anyway.

To count AI crawlers in an nginx log, which uses the combined format by default with the user agent in the sixth quote-delimited field:

awk -F'"' '{print $6}' /var/log/nginx/access.log \
  | grep -oE 'GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-CloudVertexBot|Google-Agent|Google-GeminiNotebook|Applebot|meta-externalagent|meta-webindexer|meta-externalfetcher|Amazonbot|Amzn-SearchBot|Amzn-User|Bytespider|CCBot|MistralAI-[A-Za-z]+|DuckAssistBot' \
  | sort | uniq -c | sort -rn

The match is case-sensitive on purpose. GPTBot’s user agent ends in https://openai.com/gptbot, so a case-insensitive match would count every GPTBot request twice. To see which pages one bot reads:

awk -F'"' '$6 ~ /GPTBot/ { split($2, request, " "); print request[2] }' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -20

Recording crawler reads with invisit

invisit, the tool we build, records these reads for you, and like every browser script its own tracker cannot see crawlers. @invisitdev/server records them on your server instead: invisit.handle(request) at the top of your middleware sends each crawler request to invisit in the background, robots.txt and sitemap fetches included (crawler reads). Crawler reads are never billed.

handle runs before your app, so it records every read as a 200. To see the status your server really returned, such as the 403s it sends a bot you block, call proxy before your app and crawlers after it, as the Hono guide shows:

void invisit.crawlers(request, { status: response.status })

Reads are grouped into AI training, AI search and AI assistants (the user fetchers), next to search engines, SEO tools, social previews and archivers.

invisit names every user agent in the table. Two labels differ from the operators’ wording: invisit counts CCBot and Bytespider as AI training, while Common Crawl describes an open repository and ByteDance a search crawler. invisit classifies by user agent and does not check IP lists, so a fake GPTBot counts as GPTBot; use the checks above when that matters. Beyond the table, it also recognises YouBot, AI2Bot, Diffbot, Webz.io, Timpibot, YandexAdditionalBot and the cohere-ai strings.

Checklist before you change robots.txt

• Decide per kind: block training, and allow the search crawlers and user fetchers if you want to be cited.
• Name only the bots you restrict, and leave everything else to User-agent: *.
• Add Google-Extended and Applebot-Extended for training, even though neither shows up in logs.
• Publish the file on every subdomain; robots.txt applies per host.
• If you rely on Bing’s noarchive or nocache, set the new training and grounding preferences before 1 March 2027.
• After the operator’s stated delay, check your logs: a compliant crawler still fetches /robots.txt but stops requesting the pages you disallowed.

The trap to avoid is blocking Googlebot, Applebot or bingbot to keep your pages out of AI answers. Those are search crawlers, so the block also removes you from Google Search, Apple’s search features and Bing; use the tokens and page tags above instead.