How to block AI bots on Cloudflare, Vercel, nginx and WordPress

To block AI bots, use robots.txt for the crawlers that obey it and a firewall or server rule for the ones that do not. To block training and keep search, set Training to Disallow with Bot Preference Sync on Cloudflare, deny AI Crawlers on Vercel, return 403 to training user agents in nginx or Apache, and add a robots_txt filter in WordPress. Most sites that block already made this choice: in our study of the Tranco top 20,000, 1,647 sites disallow GPTBot by name and 1,224 of them (74.3%) leave OAI-SearchBot open.

Below: what each layer can enforce, a “block training, keep search” recipe for each platform, what over-blocking costs, and how to confirm a block works. We ran the nginx, Apache and WordPress rules in containers on 8 November 2026 against 21 user agents. Cloudflare and Vercel are dashboards we can only check against their documentation, read the same day. For what every bot does, see our AI crawler reference.

What can robots.txt enforce, and what needs a firewall?

robots.txt enforces nothing. It is a published request, and RFC 9309, the standard that defines it, says its rules “are not a form of access authorization”. Cloudflare puts it the same way: “Some crawler operators may disregard your robots.txt directives”. Anything you need enforced belongs in a layer that can refuse a request.

Two of the most common opt-outs are not bots at all. Google says of Google-Extended: “Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.” Apple says the same of its token: “Applebot-Extended does not crawl webpages.” No firewall rule can match a request that does not exist, so every recipe below still needs those two lines in robots.txt.

Which AI bots ignore robots.txt?

User-triggered fetchers, by their operators’ own account, as of November 2026. OpenAI says of ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.” Perplexity says Perplexity-User “generally ignores robots.txt rules”. Google says its user-triggered fetchers, Google-Agent among them, “generally ignore robots.txt rules”. Meta says Meta-ExternalFetcher “may bypass robots.txt rules”, and Amazon says Amzn-User “may not follow all robots.txt directives”.

Two more cases matter here. ByteDance’s page for Bytespider does not mention robots.txt at all, so a rule for it has no stated commitment behind it. And a user agent match only catches bots that name themselves: a scraper sending Chrome’s user agent passes every recipe in this post. Catching those takes bot detection such as Vercel’s Bot Protection ruleset or Cloudflare’s bot products, which is a different job from opting out of AI training.

A firewall rule is the only way to stop a fetcher that ignores robots.txt. Think before you add one: each of those requests exists because someone asked an assistant about your page.

What should a “block training, keep search” rule list?

List the crawlers that collect training data and leave the search crawlers and fetchers alone. As of 8 November 2026, the names are:

Two names need a caveat. Meta says Meta-ExternalAgent is for “training foundation AI models or improving products by indexing content directly”, so it does more than training. Amazon says Amazonbot “may be used to train Amazon AI models” and also improves its products. Blocking either refuses both uses.

Keep /robots.txt readable in every firewall or server rule. A crawler that gets a 403 on that file cannot see your opt-out, and Anthropic warns that blocking its IP addresses “may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file”. The AI crawler reference has a complete robots.txt to copy, and the robots.txt generator builds one per bot.

How do I block AI bots on Cloudflare?

In the Cloudflare dashboard, set Training to Disallow, turn on Bot Preference Sync, and leave Search and Agents on Allow. The steps, from Cloudflare’s docs updated 2 November 2026:

1. Go to Security, then Settings, and filter by Bot traffic.
2. Select AI Crawler and Agent access policies.
3. Set Training to Disallow.
4. Turn on Enable Bot Preference Sync.
5. Keep Search and Agents on Allow (do not block), then save.

Use Disallow here, not Block. Cloudflare says “Training actions also apply to mixed-purpose crawlers used for both Search and Training”, so Block refuses crawlers that also feed search. With Bot Preference Sync on, Disallow “also blocks AI training crawlers at Cloudflare’s edge” and “Mixed-purpose crawlers used for both Search and Training are excluded from this edge block.” Cloudflare’s docs do not list which crawlers count as mixed-purpose. Its per-bot classification lives in BotBase, an Enterprise feature, and in its public Radar directory, which refused our requests, so we cannot name them. If you choose Block, find out first.

Disallow also writes your preference into robots.txt. Cloudflare prepends a managed block above your existing file, on all plans. The example in its docs disallows Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, and adds Content-signal: search=yes, ai-train=no, use=reference. That example does not name MistralAI-Training, so keep it in your own file if you want it covered. We counted Content-Signal lines on 762 of 10,870 sites (7.0%), and 466 use exactly that value. Cloudflare warns that Google Search Console “may occasionally report Syntax not understood” for these lines and says it has seen no impact on crawling or SEO.

To act on one crawler at a time, use AI Crawl Control, available on all plans. Blocking a crawler there creates one WAF custom rule on your zone. On the free plan it identifies crawlers by user agent only.

Three things to check on your zone:

• The default for new domains. A July 2026 changelog said new domains would block Training and Agent traffic on pages with ads from 15 September 2026. The docs updated 2 November say new domains default to Allow for all three. Read the setting instead of assuming either.
• Agents. Cloudflare “recommends Allow (do not block) for Personal Agents”.
• Bot Fight Mode. Perplexity warns that it “can’t be skipped with WAF custom rules” and can still challenge its bots even when you allow them.

How do I block AI bots on Vercel?

In Firewall, set AI Crawlers to Deny and AI Assistants to Allow. The AI Bots Managed Ruleset is on every plan and off by default. Open your project’s Firewall, then Rules, then the Configure icon in the Bot Management section, set the two categories, and publish. Vercel documents the same change for its CLI:

vercel firewall rules update ai-bots --category ai-crawlers --action deny --yes
vercel firewall rules update ai-bots --category ai-assistants --action allow --yes
vercel firewall publish

We did not run these against a live project. Vercel says a denied request gets a 403 Forbidden and “does not incur CDN Requests or Fast Data Transfer”. It defines AI Crawlers as “Bots that crawl and collect content for training” and AI Assistants as “Bots that retrieve pages in response to user actions or use for search queries”, so one switch covers both search and fetchers.

Set the category to Log before Deny, and check two things:

• Applebot. Vercel points to bots.fyi, its directory, for each bot’s category. As of 8 November 2026 it tags Applebot as both AI Crawler and Search Engine Crawler. Vercel’s docs do not say how Deny treats a bot in both groups. Apple says Applebot powers search in Spotlight, Siri and Safari, so check whether it appears in your Log results. If it does, add a custom rule that bypasses it.
• robots.txt. The docs do not say the ruleset spares /robots.txt. Vercel runs custom rules before managed rulesets, and its firewall page says a custom bypass is “allowed through any custom or managed rules”, so a bypass on that path keeps the file readable.

The ruleset has no switch inside AI Assistants, so to refuse only fetchers you need custom rules that match their user agents. Custom rules are free on all plans. If Cloudflare sits in front of Vercel, enforce in one place: Vercel says its Bot Protection ruleset does not work behind a reverse proxy.

How do I block AI bots in nginx?

Map training user agents to a flag, exclude /robots.txt, and return 403 from your server block. Put the two maps in the http context:

# Training crawlers that send their own user agent.
map $http_user_agent $ai_training_ua {
    default               0;
    "~*GPTBot"            1;
    "~*ClaudeBot"         1;
    "~*meta-externalagent" 1;
    "~*Amazonbot"         1;
    "~*MistralAI-Training" 1;
    "~*CCBot"             1;
    "~*Bytespider"        1;
}

# Keep robots.txt readable, so a crawler can still see your opt-out.
map $uri $ai_block {
    default     $ai_training_ua;
    /robots.txt 0;
}

Then, inside each server block:

if ($ai_block) {
    return 403;
}

We tested this on nginx 1.31.8 in the nginx:alpine container. nginx -t passed, and requests with realistic user agent strings behaved as intended: the seven training bots got 403 on pages and 200 on /robots.txt, while OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Applebot, bingbot, Amzn-SearchBot, Meta-ExternalFetcher, Meta-WebIndexer, MistralAI-Index and a desktop Chrome string all got 200.

Blocking the fetchers that ignore robots.txt takes one line each in the first map. These passed the same test:

    "~*ChatGPT-User"      1;
    "~*Perplexity-User"   1;
    "~*meta-externalfetcher" 1;
    "~*Amzn-User"         1;
    "~*Google-Agent"      1;

How do I block AI bots in Apache?

Add a rewrite rule above the # BEGIN WordPress block in .htaccess, or in your virtual host. Match on %{THE_REQUEST} so /robots.txt stays open:

# BEGIN AI training block
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{THE_REQUEST} !\s/robots\.txt[\s?]
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|meta-externalagent|Amazonbot|MistralAI-Training|CCBot|Bytespider) [NC]
RewriteRule ^ - [F]
</IfModule>
# END AI training block

We tested this on Apache 2.4.69 in a WordPress 7.1.4 container with pretty permalinks and WordPress’s standard rewrite rules. apache2ctl -t passed with the block inside a Directory section. The same 21 user agents gave the same results as nginx: 403 on pages for the seven training bots, 200 on /robots.txt, 200 for everyone else.

The obvious version fails. With RewriteCond %{REQUEST_URI} !^/robots\.txt$, GPTBot got a 403 on /robots.txt in our test, because WordPress serves that file through /index.php and Apache evaluates the rules again against the rewritten path. %{THE_REQUEST} holds the original request line, so it does not change.

How do I block AI bots on WordPress?

Add the training names to robots.txt with the robots_txt filter, then enforce with a server rule or your CDN if you need more than a request. WordPress core has no AI setting, and its options differ in what they block:

For a self-hosted site, this must-use plugin writes the full list and survives theme changes. Save it as wp-content/mu-plugins/ai-training-optout.php:

<?php
/* Plugin Name: AI training opt-out */
add_filter( 'robots_txt', function ( $output, $public ) {
	if ( ! $public ) {
		return $output;
	}
	$bots = array(
		'GPTBot', 'ClaudeBot', 'meta-externalagent', 'Amazonbot', 'MistralAI-Training',
		'CCBot', 'Bytespider', 'Google-Extended', 'Applebot-Extended',
	);
	$group = '';
	foreach ( $bots as $bot ) {
		$group .= "User-agent: {$bot}\n";
	}
	return $group . "Disallow: /\n\n" . $output;
}, 10, 2 );

On WordPress 7.1.4 this produced one group naming all nine, followed by WordPress’s own User-agent: * rules. Python’s urllib.robotparser, reading that output, returned blocked for GPTBot and allowed for OAI-SearchBot and Googlebot on ordinary pages. ByteDance states no robots.txt commitment, so the Bytespider line is a request with nothing behind it.

For real enforcement on self-hosted WordPress, add the Apache rule above, or the nginx rule if your host runs nginx.

What does over-blocking cost?

Over-blocking costs you the AI search and answer surfaces you wanted to keep, and at worst your search listing. The operators are direct about it, as of November 2026:

The opposite failure is blocking a name nobody uses. In our data 1,285 sites block anthropic-ai or Claude-Web by name, and 99 of those (7.7%) leave ClaudeBot, the name in use, unblocked. The full price of each block, per operator, is in Does blocking AI crawlers hurt SEO?.

How do I check the block is working?

Request a page and /robots.txt with each user agent and read the status codes. We ran this loop against the nginx config above:

url=https://example.com/
for bot in GPTBot ClaudeBot CCBot Bytespider OAI-SearchBot PerplexityBot Googlebot; do
  printf '%-14s %s\n' "$bot" "$(curl -s -o /dev/null -w '%{http_code}' -A "$bot" "$url")"
done
curl -s -o /dev/null -w 'robots as GPTBot: %{http_code}\n' -A GPTBot "${url}robots.txt"

Expect 403 for the first four, 200 for the rest, and 200 for the last line. A rule that checks IP addresses as well may treat your laptop differently from the real bot, so check your CDN’s logs too. The free AI visibility checker runs a similar test from outside your network, requesting a page as a browser and as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and compares the responses.

Compliant crawlers take time to react: OpenAI quotes about 24 hours for search, and Amazon may use a copy of robots.txt up to 30 days old. After that, your access logs should show them fetching /robots.txt and no longer requesting the pages you disallowed.

invisit, the tool we build, shows the same thing over time on the Crawlers page. If your middleware reports the real status with crawlers(request, { status }), the Responses card splits reads by status code and by crawler group, so a block appears as 403s under Training. It has two limits: a block at your CDN never reaches your server, so the SDK cannot record it, and invisit classifies by user agent without checking IP lists.

• Decide per kind: block training, keep search, and decide separately about fetchers.
• Keep Google-Extended and Applebot-Extended in robots.txt, because no firewall can match them.
• Keep /robots.txt readable by the bots you block.
• Check for a physical robots.txt on WordPress, and for Applebot on Vercel.
• Log first where the platform lets you, then deny.
• Re-check the status codes after each change, and the bot list every few months.