Block GPTBot if you do not want your content used to train OpenAI’s models, keep OAI-SearchBot open if you want to appear in ChatGPT search, and do not count on robots.txt for ChatGPT-User, because OpenAI says robots.txt rules “may not apply” to it. Most sites that make this choice split it the same way: in our scan of 10,870 readable robots.txt files on 8 November 2026, 1,647 sites block GPTBot by name and 1,224 of them (74.3%) leave OAI-SearchBot unblocked.
Below: the exact differences from OpenAI’s documentation, a decision table for docs sites, publishers, SaaS and e-commerce, a robots.txt to copy for each decision, what each block does and does not do, how long changes take, how to confirm one worked, and what other sites chose. For every other operator’s bots, see our full list of AI crawlers.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot collects content that may train OpenAI’s models, OAI-SearchBot builds what ChatGPT search shows, and ChatGPT-User opens a page when a person asks ChatGPT something. This is what OpenAI’s crawler page says about each, checked on 8 November 2026:
| GPTBot | OAI-SearchBot | ChatGPT-User | |
|---|---|---|---|
| Job | “crawl content that may be used in training our generative AI foundation models” | “surface websites in search results in ChatGPT’s search features” | “may visit a web page” when someone asks ChatGPT or a custom GPT |
| Runs on its own | Yes | Yes | No: “not used for crawling the web in an automatic fashion” |
| robots.txt | Documented tag | Documented tag | “robots.txt rules may not apply” |
| If you disallow it | “indicates a site’s content should not be used in training generative AI foundation models” | “will not be shown in ChatGPT search answers, though can still appear as navigational links” | OpenAI states no effect |
| User agent contains | GPTBot/1.4 |
OAI-SearchBot/1.4 |
ChatGPT-User/1.0 |
| Published IPs | gptbot.json, 18 prefixes | searchbot.json, 39 prefixes | chatgpt-user.json, 237 prefixes |
Three details on that page decide most of what follows.
The first is that OpenAI names only two controls: it “uses OAI-SearchBot and GPTBot robots.txt tags” for site owners. ChatGPT-User is not one of them. You can write its name in a robots.txt file, and 634 sites in our scan do, but OpenAI makes no promise that it will be read.
The second is that the two tags are separate. “Each setting is independent of the others”, and OpenAI’s own example is allowing OAI-SearchBot while disallowing GPTBot.
The third only applies when you allow both: “we may use the results from just one crawl for both use cases to avoid duplicative crawling.” Seeing little OAI-SearchBot traffic beside plenty of GPTBot traffic does not mean search is ignoring you.
OpenAI’s page also lists OAI-AdsBot, which only visits landing pages submitted as ChatGPT ads. If you do not buy those ads, it will not come up here.
Which OpenAI bots should you block for your type of site?
Most sites should allow OAI-SearchBot and ChatGPT-User and treat GPTBot as a separate decision about training. OpenAI gives no advice by site type, so this table is our judgment, not its guidance:
| Site type | GPTBot | OAI-SearchBot | ChatGPT-User | Why |
|---|---|---|---|---|
| Docs or developer tool | Allow | Allow | Allow | People ask assistants how to use your product. You want the answer to come from your current docs. |
| Publisher | Block | Allow if you want the citations | Allow, or block at the server if you sell the page | Your text is the product, so a training opt-out is the point. Search is where a citation can still happen. A licensing deal may change all three columns. |
| SaaS marketing site | Your call | Allow | Allow | Pricing and feature pages exist to be read. Blocking GPTBot costs you nothing in ChatGPT search, and allowing it costs you little on pages like these. |
| E-commerce | Block if your copy and reviews set you apart | Allow | Allow | A shopper who asks ChatGPT about a product should be able to get your page. Cart, checkout and account paths stay closed to every bot. |
Two things hold for all four rows. A disallow on GPTBot does not touch search, because the settings are independent. And private paths belong under User-agent: *, where every crawler gets them.
What robots.txt do you need for each decision?
Name only the bots you restrict and leave the rest to User-agent: *. Replace /admin/ with your own private paths. We ran each file below through a matcher that follows RFC 9309, the robots.txt standard, and checked the outcome for each bot on /, a normal page and a private page.
Allow all three. This is the docs and SaaS default. None of the three is named, so all three follow the rules for everyone.
User-agent: *
Disallow: /admin/
Disallow: /api/
Sitemap: https://example.com/sitemap.xmlBlock training, stay in ChatGPT search. The usual publisher file.
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlBlock training on a shop. The same decision, with the shopping paths closed for everyone. The /*?sort= line keeps sorted copies of category pages out, while /shoes itself stays readable.
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /*?sort=
Sitemap: https://example.com/sitemap.xmlBlock training and ChatGPT search. Live page fetches stay possible.
User-agent: GPTBot
User-agent: OAI-SearchBot
Disallow: /
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlBlock all three. The third name states your preference, and the server rule further down is what enforces it.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlKeep one section out of search. Here /members/ stays out of ChatGPT search while the rest of the site stays in it. OAI-SearchBot’s group has to repeat /admin/, for the reason in the warning below.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /admin/
Disallow: /members/
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlClosed to all crawlers except ChatGPT search. A named group replaces the * group for that bot, so OAI-SearchBot can read everything except /admin/ while every unnamed bot is refused.
User-agent: OAI-SearchBot
Disallow: /admin/
User-agent: *
Disallow: /The one setup with no good reason is blocking OAI-SearchBot and leaving GPTBot open. That gives up ChatGPT search and keeps training. Our scan found it on 2 sites. To build a file from choices instead of by hand, use the robots.txt generator.
What does each block do, and what does it not do?
Each disallow is a request to one bot about one purpose, and none of them removes your site from ChatGPT or locks a page. RFC 9309 says so directly: robots.txt rules “are not a form of access authorization”.
| Action | What it does | What it does not do |
|---|---|---|
| Disallow GPTBot | Tells OpenAI the site’s content should not be used in training | It does not affect ChatGPT search. OpenAI’s page does not say what happens to content it already collected. |
| Disallow OAI-SearchBot | Keeps your pages out of ChatGPT search answers | It does not stop you appearing as a navigational link, and it says nothing about training. |
| Disallow ChatGPT-User | States a preference | OpenAI says the rule may not apply, so it may not stop the fetch. The page also does not say whether fetched pages can be used for training. |
| Server block, ChatGPT-User | Refuses the request, so ChatGPT cannot read the page when someone asks | It does not remove you from search, which OpenAI says ChatGPT-User has no part in. |
ChatGPT search does not rest on OpenAI’s crawler alone. OpenAI’s help article on web search says it “sometimes partners with other search providers”. The crawler page does not say how a disallow on OAI-SearchBot interacts with results from those partners.
A separate kind of traffic falls outside all three rows. ChatGPT Work’s Cloud browser signs its requests with Signature-Agent: "https://chatgpt.com", and OpenAI’s crawler page does not list it. OpenAI’s article covers allowlisting it at your CDN and does not mention robots.txt.
How do you actually stop ChatGPT-User?
Refuse it at your server or CDN, because that is the only place a block does not depend on OpenAI’s choice. This nginx config returns 403 to ChatGPT-User everywhere except /robots.txt:
map $http_user_agent $block_ai_fetcher {
default 0;
~*ChatGPT-User 1;
}
server {
listen 80;
root /usr/share/nginx/html;
# Bots must still be able to read your opt-out
location = /robots.txt { }
location / {
if ($block_ai_fetcher) { return 403; }
try_files $uri $uri/ =404;
}
}We ran this in nginx with OpenAI’s documented user agent string: ChatGPT-User got 403 on pages and 200 on /robots.txt, while GPTBot, OAI-SearchBot and an ordinary browser got 200 on both. Matching the user agent is enough when you are refusing traffic, since turning away someone faking ChatGPT-User costs you nothing. Only an allow rule needs the check against OpenAI’s IP list, which our crawler guide shows how to run.
Leave /robots.txt out of any such rule. If your firewall answers 403 on that file, RFC 9309 treats it as “unavailable” and says the crawler “MAY access any resources on the server”. OpenAI’s page does not say how its bots treat that case, and if you want to be in search it recommends “allowing requests from our published IP ranges”.
How long do robots.txt changes take for OpenAI’s bots?
OpenAI gives one number: for search, “it can take ~24 hours from a site’s robots.txt update for our systems to adjust”. It gives no figure for GPTBot or ChatGPT-User. RFC 9309 says crawlers “SHOULD NOT use the cached version for more than 24 hours”, unless the file is unreachable.
We would wait two days before judging a change. OpenAI’s page is silent on content collected before you added the rule, so do not read a GPTBot disallow as a deletion.
How do you confirm the block worked?
Check three things in order: that the file says what you meant, that your server lets the bots read it, and that their requests changed afterwards.
1. Test the file. Paste it into the robots.txt generator’s “Check my file” tab, which applies the RFC rules and shows what each bot is told. Our analysis script contains the same matching logic if you want to run it yourself. Avoid Python’s urllib.robotparser for this: in our checks it departed from the RFC 27 times in 4,500 comparisons, 18 from dropping rules after a blank line and 9 from reading only the first of several User-agent: * groups.
2. Fetch the file as each bot. Bot protection can refuse a crawler that robots.txt allows, and this shows it:
for ua in GPTBot OAI-SearchBot ChatGPT-User; do
printf '%-14s %s\n' "$ua" "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://example.com/robots.txt)"
doneEach line should read 200. This only tests rules keyed on the user agent. A rule that checks OpenAI’s IP ranges treats your laptop differently from the real bot.
3. Read your logs after the waiting period. This counts each bot’s requests by robots.txt versus pages, with the status code, for an nginx or Apache combined log:
awk -F'"' 'match($6, /GPTBot|OAI-SearchBot|ChatGPT-User/) {
bot = substr($6, RSTART, RLENGTH)
split($2, request, " "); split($3, response, " ")
print bot, (request[2] == "/robots.txt" ? "robots.txt" : "pages"), response[1]
}' /var/log/nginx/access.log | sort | uniq -c | sort -rnA blocked bot that behaves shows robots.txt 200 lines and no new pages lines. A pages 403 line is your server refusing it. A pages 200 line for a bot you disallowed is worth a look: check the date of the request, then check its IP against OpenAI’s list before blaming OpenAI, because anyone can send the header. GPTBot and OAI-SearchBot may add a robots.txt marker to their user agent when fetching that file, so you can spot those fetches even in logs without paths.
No requests at all proves nothing. Bots do not visit every site every day.
Which OpenAI bots do other sites block?
Most sites block none of them by name, and those that block GPTBot usually keep search open. We read robots.txt from the Tranco top 20,000 on 8 November 2026 and got 10,870 readable files. “By name” means a group names the bot and disallows ordinary pages, and “effective” adds sites that block it only through User-agent: *.
| Bot | Sites by name | By name | Effective | Top 1,000 by name | Top 10,000 by name |
|---|---|---|---|---|---|
| GPTBot | 1,647 | 15.15% | 19.15% | 18.00% | 15.44% |
| ChatGPT-User | 634 | 5.83% | 9.83% | 9.59% | 6.69% |
| OAI-SearchBot | 416 | 3.83% | 7.84% | 7.24% | 4.61% |
Of the 1,647 GPTBot blockers, 1,224 (74.3%) leave OAI-SearchBot unblocked and 423 (25.7%) block it too. 1,032 (62.7%) leave ChatGPT-User unblocked, and 381 block all three by name. Thirteen sites shut out everyone with * and name OAI-SearchBot with an explicit allow.
More sites try to block ChatGPT-User than OAI-SearchBot: 634 against 416. That is more sites asking the one bot that may not listen, and fewer asking the one that decides ChatGPT search.
By name, on 8 November: nytimes.com, bbc.co.uk and cnn.com block all three. forbes.com blocks GPTBot and explicitly allows OAI-SearchBot. medium.com blocks GPTBot and leaves search open. reddit.com shuts out everyone with *. wikipedia.org, shopify.com and wordpress.com name none of them. The full method is in how many sites block AI crawlers.
The limits matter. Tranco ranks domains including CDNs and ad servers, 5,329 of the 20,000 did not answer, firewall blocks are invisible to a robots.txt scan, and this is one snapshot. The scan cannot sort sites by type, so nothing here tells you what publishers as a group do, beyond the sites named above.
For scale from one real site: invisit.dev’s crawler log from 4 November, when recording started, to 8 November shows GPTBot at 370 requests across 183 paths, ChatGPT-User at 51, and OAI-SearchBot at 11, of which 8 were for robots.txt. In the same window it counted 127 live assistant fetches and 0 visits with an AI referrer, and some of those fetches may be our own agents. A fetch is not a visitor, on a site this small at least.
How can you see which of the three reads your site?
Your server or CDN logs are enough, and the awk command above is free. A browser analytics script cannot see any of these bots, because they do not run it.
invisit, the tool we build, records them if you add @invisitdev/server to your middleware (crawler reads). Its Crawlers page separates AI training from AI answers and lists reads by bot, by page and by response code, so GPTBot sits apart from OAI-SearchBot and ChatGPT-User. It classifies by user agent and does not check IP lists, so a fake GPTBot counts as GPTBot. handle records every read as a 200; to see the 403s your server really sent, use proxy and crawlers as the framework guides show. The AI visibility checker is a quicker one-off: it reports what robots.txt tells each AI bot and requests your page as GPTBot and OAI-SearchBot.
Checklist before you change robots.txt
- Decide training and search separately: GPTBot for one, OAI-SearchBot for the other.
- Name only the bots you restrict, and repeat your private paths inside any named group.
- If ChatGPT-User must be stopped, refuse it at the server or CDN and keep
/robots.txtreachable. - Publish the file on every subdomain, since robots.txt applies per host.
- Test the file with an RFC 9309 parser, not
urllib.robotparser. - Wait about two days, then check your logs for
robots.txtlines without newpageslines.