How to detect AI crawlers in logs, Cloudflare, Vercel and Next.js

To detect AI crawlers, match each request’s User-Agent header on your server or CDN against the names the operators document, because crawlers do not run your analytics script. The regex below covers 33 current AI crawlers and fetchers, from GPTBot and ClaudeBot to Perplexity-User. We tested it on 8 November 2026: it matched all 32 real AI user agent strings we keep on file, none of 68 strings from other bots, and none of 16 ordinary browser and HTTP library strings.

Below: the regex and how it was built, why GA4 and script tags cannot see these requests, the trap that some AI bots look like Chrome, and copy-paste setups for nginx, Apache, Caddy JSON logs, Cloudflare, Vercel, Next.js, Express and Hono. A user agent is only a claim, so proving a request is real is its own job, covered in how to verify AI crawler IPs. For what each bot does and how to block it, see AI crawler user agents.

What regex matches every AI crawler?

This one, used case-insensitively. It is the AI training, AI search and AI assistant families from invisit’s crawler taxonomy, plus OpenAI’s OAI-AdsBot and Anthropic’s Claude-Agent, with retired names removed:

gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional

It uses only alternation and groups, with no \b, so the same string works in every engine we tried it in: GNU grep -E, mawk and BusyBox awk, nginx and Apache (PCRE), Caddy (Go RE2), JavaScript in Node and Bun, Python, and jq.

What we checked it against, on 8 November 2026:

ClaudeBot, Claude-SearchBot, Claude-User, Google-CloudVertexBot, Webz.io and Timpibot publish no full user agent string, only a name, so for those we tested the token.

What it leaves out on purpose:

• Googlebot, bingbot and Applebot. They are search crawlers that also feed AI answers, so counting them as AI would count your whole search traffic. Add them if you want them; the pillar post explains the split.
• Google-Extended and Applebot-Extended. These are robots.txt tokens and never appear in a request.
• Retired or undocumented names: anthropic-ai, Claude-Web, FacebookBot, Google-NotebookLM, cohere-ai. Anthropic’s current crawler page names only ClaudeBot, Claude-User and Claude-SearchBot, Meta’s crawler page does not name FacebookBot, Google says it supported Google-NotebookLM until August 2026, and Cohere says it does not crawl for training. In our robots.txt study, 1,559 sites still name anthropic-ai or Claude-Web, so you may meet these strings; add them if you want old or fake traffic counted.

Claude-Agent is Anthropic’s hosted browser, and it is documented separately from its crawlers. Anthropic says the token is added to a real browser’s user agent and tells you to match on Claude-Agent rather than the whole string.

Keep the regex for “is this an AI bot”. For one-off lookups, the user agent lookup tool tells you what a single string is.

Why can’t Google Analytics or a script tag see AI crawlers?

Because a script tag only reports when something runs it, and these bots read your HTML without running it. Google Analytics 4 adds a second filter: Google says “traffic from known bots and spiders is automatically excluded” and “you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded” (GA4 help, checked on 8 November 2026).

Of the operators in the regex, only Google and Apple document rendering pages at all: Google says “a headless Chromium renders the page and executes the JavaScript” (Google) and Apple says Applebot “may render the content of your website within a browser” (Apple). The crawler pages of OpenAI, Anthropic, Perplexity, Mistral, DuckDuckGo, Amazon and Meta say nothing about executing JavaScript.

The gap is real on our own site. Our crawler log for invisit.dev, in the 30 days to 8 November with recording started on 4 November, holds 127 live fetches by AI assistants and 0 visits with an AI referrer in the same window. Some of those fetches may be our own agents. The point stands: they are requests that happened and that no page-view counter would ever show.

Do AI crawlers look like Chrome?

Some do, which is the trap that breaks log filtering. Of the 32 strings we tested, 11 contain Chrome/. The ones we ran through ua-parser-js 2.0.10 came back as a normal browser:

OpenAI’s documented ChatGPT-User string has no Chrome/ in it; it ends compatible; ChatGPT-User/1.0; +https://openai.com/bot. GPTBot and Mistral’s bots share that shape. They still begin with Mozilla/5.0 AppleWebKit/537.36, so anything that treats “has a browser engine” as “human” reads them wrong too. Perplexity’s are Mozilla/5.0 (compatible; ..., and Meta’s are plain meta-externalagent/1.1 with no browser tokens at all.

Three rules that follow from it:

1. Test for bot names before you classify browsers. A parser that returns “Chrome” has not ruled out a crawler.
2. Never filter humans with grep -v Chrome, and never count “not Chrome” as bots. Both are wrong in opposite directions.
3. Do not take the Chrome version as evidence. Google, Amazon and You.com document placeholders such as Chrome/W.X.Y.Z, and the actual number is whatever the operator sends.

How do I find AI crawlers in nginx logs?

Add a map so nginx writes matching requests to their own log. The ~* prefix makes the match case-insensitive:

map $http_user_agent $ai_bot {
    default 0;
    "~*gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional" 1;
}

access_log /var/log/nginx/ai-bots.log combined if=$ai_bot;

We ran this on nginx 1.29 with eight real user agents; six matched, and the Chrome and Googlebot requests stayed out of ai-bots.log. Conditional logging with if= is documented by nginx since 1.7.0.

To count what is already in a combined-format log, Apache’s or nginx’s, this works in mawk and BusyBox awk and gives each request exactly one bot name:

RE='gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional'

awk -F'"' -v re="$RE" '{
  ua = tolower($6)
  if (match(ua, re)) count[substr(ua, RSTART, RLENGTH)]++
} END { for (bot in count) print count[bot], bot }' /var/log/nginx/access.log | sort -rn

This is the fix for the double-count the pillar post warns about: grep -oi gptbot matches twice in GPTBot/1.4; +https://openai.com/gptbot, while match() stops at the first hit. To see the IP and page behind each match, which is what you feed into verification, change the action:

awk -F'"' -v re="$RE" 'tolower($6) ~ re {
  split($1, peer, " "); split($2, request, " ")
  print peer[1], request[2]
}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

How do I find AI crawlers in Apache logs?

SetEnvIfNoCase tags the request and CustomLog ... env= writes only the tagged ones. We tested this on Apache 2.4 with mod_setenvif, which the official httpd image already loads:

SetEnvIfNoCase User-Agent "gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional" ai_bot
CustomLog logs/ai-bots.log combined env=ai_bot

The combined format puts the user agent in the same place as nginx’s, so the two awk commands above work unchanged. If you use a custom format, %{User-agent}i is how it logs the header.

How do I find AI crawlers in Caddy’s JSON logs?

Caddy’s JSON access logs store the user agent as an array at request.headers["User-Agent"], so use jq. Turn on JSON output with format json in the log directive (Caddy already defaults to JSON when stderr is not a terminal):

RE='gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional'

# Each AI bot request: client IP, path, status
jq -r --arg re "$RE" 'select(.request.headers["User-Agent"][0] | test($re; "i"))
  | [.request.client_ip, .request.uri, .status] | @tsv' access.log

# Requests per bot
jq -r --arg re "$RE" '.request.headers["User-Agent"][0] | ascii_downcase | match($re) | .string' \
  access.log | sort | uniq -c | sort -rn

In a Caddyfile you can act on the match at the edge with header_regexp, and Go’s RE2 needs the (?i) flag inline: @ai header_regexp ua User-Agent (?i)gptbot|claudebot|.... We ran that matcher with the full regex on Caddy 2 and confirmed it tags GPTBot and leaves Chrome alone.

How do I see AI crawlers on Cloudflare?

Open AI Crawl Control. It is available on all plans and shows crawlers by operator, request counts, and which of them broke your robots.txt, with no setup. We checked this section against Cloudflare’s docs on 8 November 2026 and did not run it against a live zone.

• Where: in the dashboard, under AI Crawl Control. The Metrics tab groups requests by crawler, category, operator or host, shows status codes and your most requested paths, and exports CSV. Referral data needs a paid plan.
• Accuracy depends on plan. On the free plan, AI Crawl Control “identifies AI crawlers based on their user agent strings”. Cloudflare says upgrading adds detection using its Bot Management detection ID field (docs). A spoofed GPTBot on a free zone can therefore land in the count.
• Blocking from here creates one WAF custom rule named AI Crawl Control. If a crawler you blocked still gets through, Cloudflare says to move that rule to the top of your custom rules.

Security Analytics (docs) is the other place to look. It covers all requests on every plan, keeps up to 31 days (90 on Enterprise), and its Bot analysis tab sorts traffic into Automated, Likely automated, Likely human and Verified bot. Add a filter, and use “Create custom security rule” to turn it into a WAF rule. Top statistics are sampled, and the page does not list which filter fields exist, so check the picker in your dashboard.

To match by name yourself in a WAF expression, remember that contains is case-sensitive and matches needs a Business or Enterprise plan. On any plan, with lower():

(lower(http.user_agent) contains "gptbot" or lower(http.user_agent) contains "claudebot" or lower(http.user_agent) contains "perplexitybot")

On Business or Enterprise, use the regex in one expression: lower(http.user_agent) matches "gptbot|claudebot|...". Cloudflare also exposes cf.verified_bot_category and cf.client.bot, which the next post uses to tell the real ones from the claims.

How do I see AI crawlers on Vercel?

Open Observability > AI Traffic. Vercel added it on 28 October 2026 and it reports requests per verified AI bot, the most-crawled pages, assistant retrievals, Markdown requests and errors. Observability itself is on all plans; the changelog names no plan limit for the tab.

From the terminal, vercel metrics has an AI traffic group: vercel.ai_traffic.count, .crawler.count, .assistant.count, .markdown_requests.count and .failed_requests.count. Querying those needs Observability Plus. Run vercel metrics schema vercel.ai_traffic to see the dimensions you can group by. We read these docs on 8 November 2026 but have no Vercel project to run them against.

Three places hold the raw requests:

If you need to act rather than watch, the AI bots managed ruleset can log or deny known AI bots from the Firewall, on all plans, and is off by default.

How do I detect AI crawlers in Next.js?

Use proxy.ts. Next.js 16 renamed the Middleware convention to Proxy and defaults it to the Node.js runtime (Next.js docs); the old middleware file convention is deprecated. Put the regex in one file and import it from every framework:

export const AI_BOT =
  /gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional/i

export function aiBotName(userAgent: string | null | undefined): string | null {
  return userAgent?.match(AI_BOT)?.[0].toLowerCase() ?? null
}

import { NextResponse, type NextRequest } from 'next/server'
import { aiBotName } from './ai-bots'

export function proxy(request: NextRequest) {
  const bot = aiBotName(request.headers.get('user-agent'))
  if (bot) {
    const ip = request.headers.get('x-forwarded-for')?.split(',').at(-1)?.trim()
    console.log(JSON.stringify({ bot, ip, path: request.nextUrl.pathname }))
  }
  return NextResponse.next()
}

export const config = {
  matcher: ['/((?!_next/static|_next/image).*)'],
}

Tested with next build and next start on Next.js 16.4.5: GPTBot’s requests to / and to a static public/robots.txt were both logged, and Chrome’s were not. The matcher skips only build assets on purpose, because crawlers fetch robots.txt and sitemaps too.

Two limits. Proxy runs before the route, so it cannot see the status your page answered; if you need 404s and 403s, log from a route or use an edge log. And take the IP from the last x-forwarded-for entry, the one your own proxy appended. The first entry is whatever the client sent, so a crawler faker can put anything there. Vercel overwrites the header to prevent spoofing, so on Vercel both ends are the same value.

How do I detect AI crawlers in Express and Hono?

Add one middleware before your routes and log after the response finishes, so you get the real status. Both use aiBotName from ai-bots.ts above.

import express from 'express'
import { aiBotName } from './ai-bots'

const app = express()
app.set('trust proxy', 1) // exactly one reverse proxy in front

app.use((req, res, next) => {
  const bot = aiBotName(req.get('user-agent'))
  if (bot) {
    res.on('finish', () => {
      console.log(
        JSON.stringify({ bot, ip: req.ip, path: req.originalUrl, status: res.statusCode }),
      )
    })
  }
  next()
})

import { getConnInfo } from '@hono/node-server/conninfo'
import { Hono } from 'hono'
import { aiBotName } from './ai-bots'

const app = new Hono()

app.use('*', async (c, next) => {
  await next()
  const bot = aiBotName(c.req.header('user-agent'))
  if (bot) {
    const ip =
      c.req.header('x-forwarded-for')?.split(',').at(-1)?.trim() ?? getConnInfo(c).remote.address
    console.log(JSON.stringify({ bot, ip, path: c.req.path, status: c.res.status }))
  }
})

Tested with Express 5.3.0, and Hono 4.13 with @hono/node-server 2.1, using GPTBot’s real string against / and a missing path: each logged the right status, 200 and 404, and Chrome was not logged. @hono/node-server/conninfo is the Node adapter’s helper; on other runtimes use that runtime’s getConnInfo. Express’s trust proxy of 1 is what makes req.ip the last address your proxy added; without it you get the proxy’s own address.

Which method should I use?

If a host only gives you a JavaScript snippet and no request logs, you cannot do this at all. Move detection to the CDN or to a server you control.

To keep a permanent record without running log parsing yourself: @invisitdev/server from invisit, the tool we build, sends each crawler read from your server to a Crawlers page split by AI training, AI search and assistants, by bot and by page (crawler reads). Its limits match this post’s: it classifies by user agent and does not check IP lists, so a fake GPTBot counts as GPTBot, and its taxonomy does not yet include Claude-Agent, so those hits are not recorded. The verification post covers what to do when that matters.

Checklist

• Put the regex in one shared file and test it against a line from your own log.
• Log the real client IP, not your load balancer’s, or verification will fail later.
• Keep the bot log separate from your access log, and rotate it.
• Check for bot names before any browser classification.
• Do not act on a matched name alone: verify it before you trust or allow it.
• Re-read this list when a new bot appears. Add names; the regex is a list, not a judgement.

The trap to avoid is the instinct to “exclude bots” with a browser parser. Many AI bots advertise a browser engine, and the fetchers that act for a person, such as ChatGPT-User and Perplexity-User, are the ones you most want to see.