[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt) # How to track AI crawlers on your website > Five ways to track AI crawlers: a tested access-log script, Cloudflare, Vercel, a log pipeline and invisit, with a decision table and honest limits. - Author: Miraç Berk - Published: 2026-11-08 - Page: https://invisit.dev/blog/track-ai-crawlers - Tags: ai-traffic, crawlers, server-logs ![A frosted-glass turnstile counting small matte white robots into a white doorway, one indigo counter light, on violet dithered ground.](https://invisit.dev/_astro/cover.C7i3UgUT.png) To track AI crawlers, record them where the request arrives, because most crawlers fetch your HTML without running your analytics script. You have five practical options: a script over your access log, Cloudflare AI Crawl Control, Vercel's AI Traffic tab, a log pipeline into your own database, or invisit, the tool we build. As of November 2026 the hosted dashboards keep little on their cheaper tiers: 24 hours on Cloudflare's free plan, and 12 hours on Vercel Hobby. Below: a decision table, a script we tested that counts crawlers and checks their IP addresses against the operators' own lists, a working nginx, Vector and ClickHouse setup, and the limits of each option, ours included. For what each bot is, and how to block it, see [AI crawler user agents](/blog/ai-crawlers). ## Which way of tracking AI crawlers should you use? Pick by where your site runs and whether you need to prove a crawler is genuine. Vendor details were checked on 8 November 2026 and are sourced in the sections below. | Your situation | Use | Why | | ---------------------------------------------------------------------------------------- | ------------------------------------- | -------------------------------------------------------------- | | You have a server with access logs and want an answer today | Access log script | Free, and it checks addresses against the operators' IP lists | | Your domain is already proxied by Cloudflare, and you may want to block bots | Cloudflare AI Crawl Control | No setup, block controls included, robots.txt violation report | | Your site is hosted on Vercel | Vercel AI Traffic | Already there, including errors and Markdown requests | | You need raw rows, long history and your own queries, and someone will run it | Log pipeline | You own the data, the IP field and the retention | | You want crawler history in an analytics dashboard, without running a pipeline | invisit | Reads by bot, page and status next to your visitor analytics | | You must prove a crawler is genuine | Script or pipeline; not invisit | invisit classifies by user agent only and stores no IP | | You want to block or charge crawlers | Cloudflare; not invisit | invisit reports and cannot block | | Your host gives you no server, no middleware and no logs, and no Worker can sit in front | Not invisit; ask your host for logs | invisit needs something server-side to send the reads | | You check one small site once a quarter | Script; invisit is more than you need | A single command covers it | | Option | Needs | How bots are identified | History | Can block | Cost, as of November 2026 | | ------------ | --------------------------------- | ----------------------------------------------------------- | --------------------------------------------------- | ----------------- | ------------------------------------- | | Log script | Server access logs | Your token list, plus operator IP lists where they exist | As long as you keep logs | No | Free | | Cloudflare | Domain proxied through Cloudflare | User agent on free plans; detection IDs with Bot Management | 24 hours on free plans | Yes | Included on all plans | | Vercel | Site deployed on Vercel | "Verified AI bot"; method not documented for this tab | 12 hours to 3 days; 30 days with Observability Plus | Separate firewall | Included; Plus is $1.20 per 1M events | | Log pipeline | Logs plus a database to run | Whatever you write | As long as you store it | No | Your infrastructure | | invisit | A server, Worker or log forwarder | User agent only | No retention limit | No | Free to record, up to a fair-use line | ## Why can't Google Analytics or other script-based analytics see AI crawlers? Crawlers fetch the HTML and leave, so a tracker that runs in the browser never fires. Tools also drop known bots on purpose: Google says traffic from known bots is excluded automatically in Analytics and that you "cannot disable known bot traffic exclusion or see how much known bot traffic was excluded" ([Google](https://support.google.com/analytics/answer/9888366)). [AI crawler user agents](/blog/ai-crawlers#how-can-i-see-which-ai-crawlers-visit-my-site) has more on this. Tracking therefore has to happen at the server, the CDN or the log. ## How do you track AI crawlers from an access log? Run a script over the log file. Any server that writes the standard "combined" format, which is [nginx's default](https://nginx.org/en/docs/http/ngx_http_log_module.html), already holds every crawler request with its address, path, status and user agent. This script counts each bot, separates robots.txt from page reads, counts errors, and checks each address against the list the operator publishes. ```python title="ai_crawlers.py" import collections, ipaddress, json, re, sys, urllib.request OPENAI, CLAUDE = "https://openai.com/", "https://claude.com/crawling/bots.json" PERPLEXITY = "https://www.perplexity.ai/" GOOGLE = "https://developers.google.com/static/crawling/ipranges/common-crawlers.json" BOTS = { # user agent token: (kind, the operator's published IP list, if it has one) "GPTBot": ("training", OPENAI + "gptbot.json"), "ClaudeBot": ("training", CLAUDE), "CCBot": ("training", "https://index.commoncrawl.org/ccbot.json"), "meta-externalagent": ("training", None), "Amazonbot": ("training", None), "Bytespider": ("training", None), "OAI-SearchBot": ("search", OPENAI + "searchbot.json"), "Claude-SearchBot": ("search", CLAUDE), "PerplexityBot": ("search", PERPLEXITY + "perplexitybot.json"), "ChatGPT-User": ("user fetch", OPENAI + "chatgpt-user.json"), "Claude-User": ("user fetch", CLAUDE), "Perplexity-User": ("user fetch", PERPLEXITY + "perplexity-user.json"), "DuckAssistBot": ("user fetch", "https://duckduckgo.com/duckassistbot.json"), "Googlebot": ("search engine", GOOGLE), "Applebot": ("search engine", "https://search.developer.apple.com/applebot.json"), "bingbot": ("search engine", "https://www.bing.com/toolbox/bingbot.json"), } NAME = {token.lower(): token for token in BOTS} TOKEN = re.compile("|".join(map(re.escape, BOTS)), re.I) LINE = re.compile(r'(\S+) \S+ \S+ \[[^]]+\] "\S+ (\S+)[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"') def load(url, cache={}): """The operator's networks, or None if the list could not be read.""" if url not in cache: try: request = urllib.request.Request(url, headers={"User-Agent": "ip-check"}) prefixes = json.load(urllib.request.urlopen(request, timeout=20))["prefixes"] cache[url] = [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"]) for p in prefixes] except (OSError, ValueError, KeyError): cache[url] = None return cache[url] def listed(ip, networks): try: address = ipaddress.ip_address(ip) except ValueError: return False return any(address in net for net in networks if net.version == address.version) rows = collections.defaultdict(collections.Counter) paths = collections.Counter() focus = sys.argv[1].lower() if len(sys.argv) > 1 else None for line in sys.stdin: match = LINE.match(line) found = match and TOKEN.search(match[4]) if not found: continue ip, path, status, bot = match[1], match[2].split("?")[0], int(match[3]), NAME[found[0].lower()] row = rows[bot] row["reads"] += 1 row["robots"] += path == "/robots.txt" row["errors"] += status >= 400 url = BOTS[bot][1] networks = load(url) if url else None row["checked"] += networks is not None row["fake"] += networks is not None and not listed(ip, networks) if bot.lower() == focus: paths[path] += 1 if focus: for path, count in paths.most_common(20): print(f"{count:>7} {path}") sys.exit() print(f"{'bot':<20}{'kind':<15}{'reads':>7}{'robots':>8}{'4xx/5xx':>9}{'not in IP list':>16}") for bot, row in sorted(rows.items(), key=lambda item: -item[1]["reads"]): fake = row["fake"] if row["checked"] else "no list" print(f"{bot:<20}{BOTS[bot][0]:<15}{row['reads']:>7}{row['robots']:>8}{row['errors']:>9}{fake:>16}") ``` ```sh title="Terminal" zcat -f /var/log/nginx/access.log* | python3 ai_crawlers.py # all bots zcat -f /var/log/nginx/access.log* | python3 ai_crawlers.py gptbot # one bot's top pages ``` We tested it on 8 November 2026 with Python 3 against a small log we wrote using the user agents the operators publish, real addresses taken from their lists, and one GPTBot line sent from a documentation-range address. It counted the fake: ```text title="Output, on our test log" bot kind reads robots 4xx/5xx not in IP list GPTBot training 5 1 0 1 ClaudeBot training 3 0 1 0 ChatGPT-User user fetch 2 0 1 0 OAI-SearchBot search 2 1 0 0 Claude-User user fetch 1 0 0 0 PerplexityBot search 1 0 0 0 meta-externalagent training 1 0 0 no list Googlebot search engine 1 1 0 0 ``` Reading the columns: - **robots** is how many requests were for `/robots.txt`. A bot with nothing but robots.txt fetches has checked your rules and read little. - **4xx/5xx** points at the pages crawlers failed on. - **not in IP list** counts requests whose address is not on the operator's published list. The script reads each list fresh on every run, and when a list cannot be fetched it marks that bot "no list" instead of calling everything fake. "No list" also means the script has no JSON list for that bot: Amazon and ByteDance publish ranges on web pages the script does not parse, and Meta's crawler page did not load when we checked on 8 November (on 3 November it published no list). The limits are the ones logs always have. If a CDN or load balancer sits in front of your server, the address field holds its IP, and "not in IP list" means nothing until the real client address is logged. If the CDN answers from cache, that request never reaches the origin's log. A host that gives you no access logs rules this option out. And a listed address is only as good as the list: operators add ranges, so a one-off miss is a reason to look closer, not proof of a fake. ## How do you track AI crawlers with Cloudflare AI Crawl Control? If your domain is proxied through Cloudflare, open AI Crawl Control in the dashboard; there is nothing to install. Cloudflare [says](https://developers.cloudflare.com/ai-crawl-control/) it is available on all plans and works with "zero configuration", and the only [prerequisite](https://developers.cloudflare.com/ai-crawl-control/get-started/) is traffic flowing through its proxy. What you get, per [Cloudflare's docs](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/) as of November 2026: - Requests by crawler, operator, category, host and path, with status codes, data transfer and CSV export. - A [robots.txt violations table](https://developers.cloudflare.com/ai-crawl-control/features/track-robots-txt/), showing crawlers that requested paths you disallowed. Cloudflare warns it compares past requests with your current rules, so a newly added `Disallow` flags requests that were fine at the time. - [Allow and block controls](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) per crawler, enforced with a WAF custom rule. A custom block response needs a paid plan, and pay per crawl is in private beta. - Referrals from AI platforms, on paid plans only. Two limits matter. History is short: the get-started page says "on free plans, the Metrics tab only displays metrics for the past 24 hours", and its plan table lists a "maximum 24-hour analytics window" under all plans, with configurable timeframes only for Enterprise with Bot Management. Identification is plan-bound too: "On the free plan, AI Crawl Control identifies AI crawlers based on their user agent strings", and the stronger detection ID field needs Bot Management. So on a free plan, Cloudflare's identification rests on the user agent, as invisit's does. Cloudflare also exposes the same data through its GraphQL Analytics API, if you want to export it yourself. ## How do you track AI crawlers with Vercel AI Traffic? If you host on Vercel, open Observability, then AI Traffic. Vercel added the tab on [28 October 2026](https://vercel.com/changelog/ai-traffic-insights-now-available-in-vercel-observability), and it counts AI bots requesting your pages, split into crawlers that index or train and assistants fetching for a user. That is a different measurement from people clicking through. The [tab](https://vercel.com/docs/observability/insights#ai-traffic) shows the paths crawlers and assistants request most, paths where AI bots received an error, and paths requested as Markdown. Vercel's changelog calls these "requests by verified AI bot"; its pages do not say how this tab decides a bot is verified. Observability itself is [available on all plans](https://vercel.com/docs/observability), and none of the pages we read states a plan limit for this tab. The limit is retention. Vercel's [limits table](https://vercel.com/docs/observability/observability-plus) gives Observability's data retention as 12 hours on Hobby, 1 day on Pro and 3 days on Enterprise, against 30 days with Observability Plus, and it does not list AI Traffic separately. Plus costs $1.20 per million events and is on by default for teams created or upgraded to Pro on or after 3 April 2026. Querying these metrics with `vercel metrics` from the CLI also needs Plus. It only sees what runs on Vercel. To act on what you find, Vercel points to [WAF custom rules](https://vercel.com/docs/bot-management), a separate product from this report. ## How do you build a log pipeline for AI crawlers? Send only the crawler lines into a database you query with SQL. This is the right call when you already run log infrastructure or need raw rows with IPs for months; if neither is true, the script above is enough. This setup ran end to end in containers on 8 November 2026, with nginx, Vector 0.50 and ClickHouse 26.3. nginx writes a JSON line only for requests whose user agent matches, using the `if=` parameter of [`access_log`](https://nginx.org/en/docs/http/ngx_http_log_module.html), so ordinary traffic adds nothing to this file: ```nginx title="nginx.conf" map $http_user_agent $ai_crawler { default 0; "~*GPTBot|ClaudeBot|OAI-SearchBot|ChatGPT-User|Claude-User|PerplexityBot" 1; } log_format crawler escape=json '{"time":"$time_iso8601","ip":"$remote_addr","host":"$host","method":"$request_method",' '"uri":"$request_uri","status":$status,"bytes":$body_bytes_sent,"ua":"$http_user_agent"}'; access_log /var/log/nginx/crawlers.jsonl crawler if=$ai_crawler; ``` The `map` and `log_format` go in the `http` block and the `access_log` line in your `server` block. An `access_log` in a `server` block replaces any log inherited from `http`, which we confirmed in a separate test, so repeat your normal `access_log` line there too. [Vector](https://vector.dev/docs/reference/configuration/sinks/clickhouse/) tails that file and writes to ClickHouse: ```yaml title="vector.yaml" sources: nginx: type: file include: [/var/log/nginx/crawlers.jsonl] read_from: beginning transforms: parsed: type: remap inputs: [nginx] source: | . = parse_json!(.message) sinks: clickhouse: type: clickhouse inputs: [parsed] endpoint: http://clickhouse:8123 table: crawler_hits date_time_best_effort: true skip_unknown_fields: true ``` ```sql title="ClickHouse" CREATE TABLE crawler_hits ( time DateTime, ip String, host LowCardinality(String), method LowCardinality(String), uri String, status UInt16, bytes UInt64, ua String ) ENGINE = MergeTree ORDER BY time; SELECT extract(ua, '(?i)(GPTBot|ClaudeBot|OAI-SearchBot|ChatGPT-User|Claude-User|PerplexityBot)') AS bot, count() AS reads, countIf(path(uri) = '/robots.txt') AS robots, countIf(status >= 400) AS errors, uniqExact(path(uri)) AS pages FROM crawler_hits WHERE time > now() - INTERVAL 30 DAY GROUP BY bot ORDER BY reads DESC; ``` Hosted platforms can feed the same kind of table, though we checked these two only against their docs: - **Cloudflare [Logpush](https://developers.cloudflare.com/logs/logpush/)** is listed as available on all plans, subject to the datasets your account has. Its [`http_requests`](https://developers.cloudflare.com/logs/reference/log-fields/zone/http_requests/) dataset carries `ClientIP`, `ClientRequestPath` and `ClientRequestUserAgent`, pushed to storage you choose. Each account includes 25 GB a month, then [$0.03 per GB](https://developers.cloudflare.com/logs/logpush/pricing/) to R2 or $0.10 per GB to other destinations, as of November 2026. It "cannot backfill historical data", so a failed job loses those logs. - **[Vercel Log Drains](https://vercel.com/docs/drains)** are on Pro and Enterprise at $0.50 per GB, and their [schema](https://vercel.com/docs/drains/reference/logs) carries the user agent and client IP under `proxy`. A pipeline gives you IP verification, as long as you keep the address and join it against the operators' lists. It also gives you a system to maintain, which is the honest cost of this option. ## How does invisit track AI crawlers, and where does it fall short? invisit records crawler reads from something that runs server-side and shows them on a Crawlers page. [`@invisitdev/server`](https://invisit.dev/docs/server/sdk.md#crawler-reads) calls it from your middleware in [most server frameworks](https://invisit.dev/docs/server/frameworks.md); a Cloudflare Worker routed in front of any origin does the same for static sites; and the dashboard's Connect server panel also has a Node forwarder for nginx or Caddy access logs. We run the Worker version on invisit.dev, and [what it recorded](/blog/ai-crawlers-new-website) is public. The page shows reads, AI share, assistant fetches and search share, a chart over time, and the bots, pages, categories and response codes behind them, with tabs for training, AI answers, search and other. Clicking a bot lists the pages it read, and clicking an error code lists the pages that returned it. Reads are kept with the rest of your analytics, with no retention window. The limits, plainly: - **User agent only, no IP check.** The collector classifies each read from the user agent and stores no address, so a fake GPTBot counts as GPTBot. Use the script above or the pipeline when that matters. It recognises 87 named crawlers, plus a catch-all that files any other user agent that calls itself a bot under Other. - **It needs a server, Worker or log forwarder.** With none of those there is nothing to send the reads, and a page your CDN answers from cache never reaches middleware at your origin. If you cache HTML, put the recording at the edge. - **Status depends on placement.** `handle` runs before your app and records every read as 200. Use `proxy` before and `crawlers` after to get the real status. - **It reports and does not act.** There is no block control and no robots.txt violation report; Cloudflare has both. - **Fair use.** Crawler reads are not billed, up to a [line of three times the plan's events](https://invisit.dev/docs/guides/billing.md#what-counts-as-an-event): 60,000 reads a month on the free plan. Past it, reads stop being recorded until the next month, while visitor analytics carry on. A site with far more crawl than visitors can reach that, and a pipeline has no such line. - **Limited backfill.** A forwarder can send lines at most 72 hours old, so history starts close to the day you connect. ## What should you check once tracking is running? Once reads are arriving, these checks turn them into decisions. Use the [user agent lookup](/tools/user-agent-lookup) for anything you do not recognise, and the [AI visibility tool](/tools/ai-visibility) to see how your site looks to these bots. - [ ] Split reads by kind. Training, search and user fetches mean different things and call for different robots.txt rules. - [ ] Compare robots.txt fetches with page reads for each bot, then check that crawlers you disallowed stopped reading pages after the operator's stated delay. - [ ] List the 4xx and 5xx pages for assistants. Each one is an answer that could not use your page. - [ ] Check addresses for the bots you act on, using the operator's list, before you trust a spike. - [ ] Put assistant fetches next to [visits from AI assistants](/blog/ai-assistant-traffic). The first shows demand, the second shows clicks. - [ ] Compare your setup with [what other sites block](/blog/how-many-sites-block-ai-crawlers), then decide with the [robots.txt generator](/tools/robots-txt-generator). The trap to avoid is reading user agent counts as proof. Anyone can send `GPTBot` in a header, so count with whatever you have, and verify before you block or report a number you will be held to.