[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt) # How to detect AI crawlers in logs, Cloudflare, Vercel and Next.js > One regex for GPTBot, ClaudeBot and every current AI bot, with tested configs for nginx, Apache, Caddy, Next.js, Express and Hono, plus Cloudflare and Vercel. - Author: Miraç Berk - Published: 2026-11-08 - Page: https://invisit.dev/blog/detect-ai-crawlers - Tags: ai-traffic, crawlers, server-logs ![A frosted-glass magnifier resting on a white ledger of log lines, one row glowing indigo, over violet square-pixel dither in soft light.](https://invisit.dev/_astro/cover.aWDIuDS_.png) To detect AI crawlers, match each request's `User-Agent` header on your server or CDN against the names the operators document, because crawlers do not run your analytics script. The regex below covers 33 current AI crawlers and fetchers, from GPTBot and ClaudeBot to Perplexity-User. We tested it on 8 November 2026: it matched all 32 real AI user agent strings we keep on file, none of 68 strings from other bots, and none of 16 ordinary browser and HTTP library strings. Below: the regex and how it was built, why GA4 and script tags cannot see these requests, the trap that some AI bots look like Chrome, and copy-paste setups for nginx, Apache, Caddy JSON logs, Cloudflare, Vercel, Next.js, Express and Hono. A user agent is only a claim, so proving a request is real is its own job, covered in [how to verify AI crawler IPs](/blog/verify-ai-crawler-ip). For what each bot does and how to block it, see [AI crawler user agents](/blog/ai-crawlers). ## What regex matches every AI crawler? This one, used case-insensitively. It is the AI training, AI search and AI assistant families from invisit's [crawler taxonomy](/bots), plus OpenAI's OAI-AdsBot and Anthropic's Claude-Agent, with retired names removed: ```text title="ai-bots.regex" gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional ``` It uses only alternation and groups, with no `\b`, so the same string works in every engine we tried it in: GNU `grep -E`, mawk and BusyBox awk, nginx and Apache (PCRE), Caddy (Go RE2), JavaScript in Node and Bun, Python, and jq. What we checked it against, on 8 November 2026: | Set | Strings | Matched | | ------------------------------------------------------------ | ------- | ------- | | AI bots' user agent strings, as published by their operators | 32 | 32 | | Other bots: Googlebot, bingbot, Applebot, Slackbot and more | 68 | 0 | | Chrome, Firefox, Safari, Edge and Samsung Internet | 10 | 0 | | curl, python-requests, Go, axios, node, HeadlessChrome | 6 | 0 | ClaudeBot, Claude-SearchBot, Claude-User, Google-CloudVertexBot, Webz.io and Timpibot publish no full user agent string, only a name, so for those we tested the token. What it leaves out on purpose: - **Googlebot, bingbot and Applebot.** They are search crawlers that also feed AI answers, so counting them as AI would count your whole search traffic. Add them if you want them; [the pillar post](/blog/ai-crawlers) explains the split. - **Google-Extended and Applebot-Extended.** These are robots.txt tokens and never appear in a request. - **Retired or undocumented names:** `anthropic-ai`, `Claude-Web`, `FacebookBot`, `Google-NotebookLM`, `cohere-ai`. Anthropic's [current crawler page](https://support.claude.com/en/articles/8896518) names only ClaudeBot, Claude-User and Claude-SearchBot, Meta's [crawler page](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) does not name FacebookBot, Google says it [supported Google-NotebookLM until August 2026](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), and Cohere [says it does not crawl for training](https://docs.cohere.com/docs/cohere-web-crawlers). In our [robots.txt study](/blog/how-many-sites-block-ai-crawlers), 1,559 sites still name `anthropic-ai` or `Claude-Web`, so you may meet these strings; add them if you want old or fake traffic counted. Claude-Agent is Anthropic's hosted browser, and it is [documented separately](https://support.claude.com/en/articles/18234133-how-do-anthropic-hosted-browsers-identify-themselves-when-they-browse-the-web-on-a-user-s-behalf) from its crawlers. Anthropic says the token is added to a real browser's user agent and tells you to match on `Claude-Agent` rather than the whole string. Keep the regex for "is this an AI bot". For one-off lookups, the [user agent lookup tool](/tools/user-agent-lookup) tells you what a single string is. ## Why can't Google Analytics or a script tag see AI crawlers? Because a script tag only reports when something runs it, and these bots read your HTML without running it. Google Analytics 4 adds a second filter: Google says "traffic from known bots and spiders is automatically excluded" and "you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded" ([GA4 help](https://support.google.com/analytics/answer/9888366), checked on 8 November 2026). Of the operators in the regex, only Google and Apple document rendering pages at all: Google says "a headless Chromium renders the page and executes the JavaScript" ([Google](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)) and Apple says Applebot "may render the content of your website within a browser" ([Apple](https://support.apple.com/en-us/119829)). The crawler pages of OpenAI, Anthropic, Perplexity, Mistral, DuckDuckGo, Amazon and Meta say nothing about executing JavaScript. The gap is real on our own site. Our crawler log for invisit.dev, in the 30 days to 8 November with recording started on 4 November, holds 127 live fetches by AI assistants and 0 visits with an AI referrer in the same window. Some of those fetches may be our own agents. The point stands: they are requests that happened and that no page-view counter would ever show. ## Do AI crawlers look like Chrome? Some do, which is the trap that breaks log filtering. Of the 32 strings we tested, 11 contain `Chrome/`. The ones we ran through [`ua-parser-js`](https://www.npmjs.com/package/ua-parser-js) 2.0.10 came back as a normal browser: | Bot | The string starts or ends like | `ua-parser-js` says | | --------------------- | ----------------------------------------------------------- | ------------------------ | | OAI-SearchBot | Chrome 131 on macOS, then `; compatible; OAI-SearchBot/1.4` | Chrome 131 on macOS | | Google-Agent | Mobile Chrome on Android, then `Google-Agent` | Mobile Chrome on Android | | Google-GeminiNotebook | Mobile Chrome on Android, then the token | Mobile Chrome on Android | | Amazonbot, Amzn-User | Chrome, with `compatible; Amazonbot/0.1` inside | Chrome | | YouBot | Chrome, with `compatible; YouBot/1.0` inside | Chrome | OpenAI's documented [ChatGPT-User string](https://platform.openai.com/docs/bots) has no `Chrome/` in it; it ends `compatible; ChatGPT-User/1.0; +https://openai.com/bot`. GPTBot and Mistral's bots share that shape. They still begin with `Mozilla/5.0 AppleWebKit/537.36`, so anything that treats "has a browser engine" as "human" reads them wrong too. Perplexity's are `Mozilla/5.0 (compatible; ...`, and Meta's are plain `meta-externalagent/1.1` with no browser tokens at all. Three rules that follow from it: 1. Test for bot names before you classify browsers. A parser that returns "Chrome" has not ruled out a crawler. 2. Never filter humans with `grep -v Chrome`, and never count "not Chrome" as bots. Both are wrong in opposite directions. 3. Do not take the Chrome version as evidence. Google, Amazon and You.com document placeholders such as `Chrome/W.X.Y.Z`, and the actual number is whatever the operator sends. ## How do I find AI crawlers in nginx logs? Add a `map` so nginx writes matching requests to their own log. The `~*` prefix makes the match case-insensitive: ```nginx title="/etc/nginx/conf.d/ai-bots.conf" map $http_user_agent $ai_bot { default 0; "~*gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional" 1; } access_log /var/log/nginx/ai-bots.log combined if=$ai_bot; ``` We ran this on nginx 1.29 with eight real user agents; six matched, and the Chrome and Googlebot requests stayed out of `ai-bots.log`. Conditional logging with `if=` is [documented by nginx](https://nginx.org/en/docs/http/ngx_http_log_module.html) since 1.7.0. To count what is already in a combined-format log, Apache's or nginx's, this works in mawk and BusyBox awk and gives each request exactly one bot name: ```sh title="count-ai-bots.sh" RE='gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional' awk -F'"' -v re="$RE" '{ ua = tolower($6) if (match(ua, re)) count[substr(ua, RSTART, RLENGTH)]++ } END { for (bot in count) print count[bot], bot }' /var/log/nginx/access.log | sort -rn ``` This is the fix for the double-count [the pillar post warns about](/blog/ai-crawlers#how-can-i-see-which-ai-crawlers-visit-my-site): `grep -oi gptbot` matches twice in `GPTBot/1.4; +https://openai.com/gptbot`, while `match()` stops at the first hit. To see the IP and page behind each match, which is what you feed into verification, change the action: ```sh title="ai-bot-ips.sh" awk -F'"' -v re="$RE" 'tolower($6) ~ re { split($1, peer, " "); split($2, request, " ") print peer[1], request[2] }' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20 ``` > [!WARNING] > If nginx or Apache sits behind a load balancer, `$1` is that balancer's address. Log the real client address first ([`set_real_ip_from`](https://nginx.org/en/docs/http/ngx_http_realip_module.html) in nginx, [`mod_remoteip`](https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html) in Apache), or every crawler will look unverified. ## How do I find AI crawlers in Apache logs? `SetEnvIfNoCase` tags the request and `CustomLog ... env=` writes only the tagged ones. We tested this on Apache 2.4 with [`mod_setenvif`](https://httpd.apache.org/docs/2.4/mod/mod_setenvif.html), which the official `httpd` image already loads: ```apache title="ai-bots.conf" SetEnvIfNoCase User-Agent "gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional" ai_bot CustomLog logs/ai-bots.log combined env=ai_bot ``` The `combined` format puts the user agent in the same place as nginx's, so the two `awk` commands above work unchanged. If you use a custom format, [`%{User-agent}i`](https://httpd.apache.org/docs/2.4/mod/mod_log_config.html) is how it logs the header. ## How do I find AI crawlers in Caddy's JSON logs? Caddy's JSON access logs store the user agent as an array at `request.headers["User-Agent"]`, so use `jq`. Turn on JSON output with `format json` in the [`log` directive](https://caddyserver.com/docs/caddyfile/directives/log) (Caddy already defaults to JSON when `stderr` is not a terminal): ```sh title="caddy-ai-bots.sh" RE='gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional' # Each AI bot request: client IP, path, status jq -r --arg re "$RE" 'select(.request.headers["User-Agent"][0] | test($re; "i")) | [.request.client_ip, .request.uri, .status] | @tsv' access.log # Requests per bot jq -r --arg re "$RE" '.request.headers["User-Agent"][0] | ascii_downcase | match($re) | .string' \ access.log | sort | uniq -c | sort -rn ``` In a Caddyfile you can act on the match at the edge with `header_regexp`, and Go's RE2 needs the `(?i)` flag inline: `@ai header_regexp ua User-Agent (?i)gptbot|claudebot|...`. We ran that matcher with the full regex on Caddy 2 and confirmed it tags GPTBot and leaves Chrome alone. ## How do I see AI crawlers on Cloudflare? Open AI Crawl Control. It is [available on all plans](https://developers.cloudflare.com/ai-crawl-control/) and shows crawlers by operator, request counts, and which of them broke your robots.txt, with no setup. We checked this section against Cloudflare's docs on 8 November 2026 and did not run it against a live zone. - **Where:** in the dashboard, under AI Crawl Control. The [Metrics tab](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/) groups requests by crawler, category, operator or host, shows status codes and your most requested paths, and exports CSV. Referral data needs a paid plan. - **Accuracy depends on plan.** On the free plan, AI Crawl Control "identifies AI crawlers based on their user agent strings". Cloudflare says upgrading adds detection using its Bot Management detection ID field ([docs](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/)). A spoofed GPTBot on a free zone can therefore land in the count. - **Blocking from here** creates [one WAF custom rule](https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/) named AI Crawl Control. If a crawler you blocked still gets through, Cloudflare says to move that rule to the top of your custom rules. **Security Analytics** ([docs](https://developers.cloudflare.com/waf/analytics/security-analytics/)) is the other place to look. It covers all requests on every plan, keeps up to 31 days (90 on Enterprise), and its Bot analysis tab sorts traffic into Automated, Likely automated, Likely human and Verified bot. Add a filter, and use "Create custom security rule" to turn it into a WAF rule. Top statistics are sampled, and the page does not list which filter fields exist, so check the picker in your dashboard. To match by name yourself in a WAF expression, remember that `contains` is [case-sensitive and `matches` needs a Business or Enterprise plan](https://developers.cloudflare.com/ruleset-engine/rules-language/operators/). On any plan, with `lower()`: ```text title="WAF expression, any plan" (lower(http.user_agent) contains "gptbot" or lower(http.user_agent) contains "claudebot" or lower(http.user_agent) contains "perplexitybot") ``` On Business or Enterprise, use the regex in one expression: `lower(http.user_agent) matches "gptbot|claudebot|..."`. Cloudflare also exposes `cf.verified_bot_category` and `cf.client.bot`, which the [next post](/blog/verify-ai-crawler-ip#how-does-cloudflare-verify-bots) uses to tell the real ones from the claims. ## How do I see AI crawlers on Vercel? Open Observability > AI Traffic. Vercel [added it on 28 October 2026](https://vercel.com/changelog/ai-traffic-insights-now-available-in-vercel-observability) and it reports requests per verified AI bot, the most-crawled pages, assistant retrievals, Markdown requests and errors. [Observability](https://vercel.com/docs/observability) itself is on all plans; the changelog names no plan limit for the tab. From the terminal, `vercel metrics` has an [AI traffic group](https://vercel.com/docs/cli/metrics): `vercel.ai_traffic.count`, `.crawler.count`, `.assistant.count`, `.markdown_requests.count` and `.failed_requests.count`. Querying those needs Observability Plus. Run `vercel metrics schema vercel.ai_traffic` to see the dimensions you can group by. We read these docs on 8 November 2026 but have no Vercel project to run them against. Three places hold the raw requests: | Tool | What it covers | User agent available? | | ----------------------------------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------ | | [Runtime Logs](https://vercel.com/docs/logs/runtime) | Functions and Routing Middleware; static files only when served from cache | Your own logs, if you print it | | [`vercel logs requests`](https://vercel.com/docs/cli/logs) | CDN request records, filterable with KQL | Run `vercel logs requests --help` for the fields | | [Log Drains](https://vercel.com/docs/drains/reference/logs) | Function, edge, static and other sources, sent to your own endpoint | Yes: `proxy.userAgent` and `proxy.clientIp` | If you need to act rather than watch, the [AI bots managed ruleset](https://vercel.com/docs/bot-management) can log or deny known AI bots from the Firewall, on all plans, and is off by default. ## How do I detect AI crawlers in Next.js? Use `proxy.ts`. Next.js 16 renamed the Middleware convention to Proxy and defaults it to the Node.js runtime ([Next.js docs](https://nextjs.org/docs/app/api-reference/file-conventions/proxy)); the old `middleware` file convention is deprecated. Put the regex in one file and import it from every framework: ```ts title="ai-bots.ts" export const AI_BOT = /gptbot|oai-searchbot|oai-adsbot|chatgpt-user|claudebot|claude-searchbot|claude-user|claude-agent|perplexitybot|perplexity-user|mistralai-(training|index|user)|duckassistbot|meta-external(agent|fetcher)|meta-webindexer|google-agent|google-gemininotebook|google-cloudvertexbot|amazonbot|amzn-(searchbot|user)|ccbot|bytespider|youbot|exasearchbot|exauser|firecrawlagent|ai2bot|webzio|timpibot|yandexadditional/i export function aiBotName(userAgent: string | null | undefined): string | null { return userAgent?.match(AI_BOT)?.[0].toLowerCase() ?? null } ``` ```ts title="proxy.ts" import { NextResponse, type NextRequest } from 'next/server' import { aiBotName } from './ai-bots' export function proxy(request: NextRequest) { const bot = aiBotName(request.headers.get('user-agent')) if (bot) { const ip = request.headers.get('x-forwarded-for')?.split(',').at(-1)?.trim() console.log(JSON.stringify({ bot, ip, path: request.nextUrl.pathname })) } return NextResponse.next() } export const config = { matcher: ['/((?!_next/static|_next/image).*)'], } ``` Tested with `next build` and `next start` on Next.js 16.4.5: GPTBot's requests to `/` and to a static `public/robots.txt` were both logged, and Chrome's were not. The matcher skips only build assets on purpose, because crawlers fetch `robots.txt` and sitemaps too. Two limits. Proxy runs before the route, so it cannot see the status your page answered; if you need 404s and 403s, log from a route or use an edge log. And take the IP from the last `x-forwarded-for` entry, the one your own proxy appended. The first entry is whatever the client sent, so a crawler faker can put anything there. Vercel [overwrites the header](https://vercel.com/docs/headers/request-headers) to prevent spoofing, so on Vercel both ends are the same value. ## How do I detect AI crawlers in Express and Hono? Add one middleware before your routes and log after the response finishes, so you get the real status. Both use `aiBotName` from `ai-bots.ts` above. ```ts title="express.ts" import express from 'express' import { aiBotName } from './ai-bots' const app = express() app.set('trust proxy', 1) // exactly one reverse proxy in front app.use((req, res, next) => { const bot = aiBotName(req.get('user-agent')) if (bot) { res.on('finish', () => { console.log( JSON.stringify({ bot, ip: req.ip, path: req.originalUrl, status: res.statusCode }), ) }) } next() }) ``` ```ts title="hono.ts" import { getConnInfo } from '@hono/node-server/conninfo' import { Hono } from 'hono' import { aiBotName } from './ai-bots' const app = new Hono() app.use('*', async (c, next) => { await next() const bot = aiBotName(c.req.header('user-agent')) if (bot) { const ip = c.req.header('x-forwarded-for')?.split(',').at(-1)?.trim() ?? getConnInfo(c).remote.address console.log(JSON.stringify({ bot, ip, path: c.req.path, status: c.res.status })) } }) ``` Tested with Express 5.3.0, and Hono 4.13 with `@hono/node-server` 2.1, using GPTBot's real string against `/` and a missing path: each logged the right status, 200 and 404, and Chrome was not logged. `@hono/node-server/conninfo` is the Node adapter's helper; on other runtimes use that runtime's `getConnInfo`. Express's `trust proxy` of `1` is what makes `req.ip` the last address your proxy added; without it you get the proxy's own address. ## Which method should I use? | Where you run | Fastest way to see AI crawlers | What it cannot tell you | | -------------------------- | ---------------------------------------------------------- | -------------------------------------------------- | | A VPS with nginx or Apache | The `map` or `SetEnvIfNoCase` log, then `count-ai-bots.sh` | Whether the name is honest | | Caddy | `jq` on the JSON log | Whether the name is honest | | Behind Cloudflare | AI Crawl Control | On the free plan, whether the name is honest | | On Vercel | Observability > AI Traffic | The raw requests; use Log Drains for those | | Next.js, Express, Hono | One middleware calling `aiBotName` | Static files your CDN serves before your code runs | If a host only gives you a JavaScript snippet and no request logs, you cannot do this at all. Move detection to the CDN or to a server you control. To keep a permanent record without running log parsing yourself: `@invisitdev/server` from invisit, the tool we build, sends each crawler read from your server to a Crawlers page split by AI training, AI search and assistants, by bot and by page ([crawler reads](https://invisit.dev/docs/server/sdk.md#crawler-reads)). Its limits match this post's: it classifies by user agent and does not check IP lists, so a fake GPTBot counts as GPTBot, and its taxonomy does not yet include Claude-Agent, so those hits are not recorded. The verification post covers what to do when that matters. ## Checklist - [ ] Put the regex in one shared file and test it against a line from your own log. - [ ] Log the real client IP, not your load balancer's, or verification will fail later. - [ ] Keep the bot log separate from your access log, and rotate it. - [ ] Check for bot names before any browser classification. - [ ] Do not act on a matched name alone: [verify it](/blog/verify-ai-crawler-ip) before you trust or allow it. - [ ] Re-read this list when a new bot appears. Add names; the regex is a list, not a judgement. The trap to avoid is the instinct to "exclude bots" with a browser parser. Many AI bots advertise a browser engine, and the fetchers that act for a person, such as ChatGPT-User and Perplexity-User, are the ones you most want to see.