How to track AI crawlers on your website

To track AI crawlers, record them where the request arrives, because most crawlers fetch your HTML without running your analytics script. You have five practical options: a script over your access log, Cloudflare AI Crawl Control, Vercel’s AI Traffic tab, a log pipeline into your own database, or invisit, the tool we build. As of November 2026 the hosted dashboards keep little on their cheaper tiers: 24 hours on Cloudflare’s free plan, and 12 hours on Vercel Hobby.

Below: a decision table, a script we tested that counts crawlers and checks their IP addresses against the operators’ own lists, a working nginx, Vector and ClickHouse setup, and the limits of each option, ours included. For what each bot is, and how to block it, see AI crawler user agents.

Which way of tracking AI crawlers should you use?

Pick by where your site runs and whether you need to prove a crawler is genuine. Vendor details were checked on 8 November 2026 and are sourced in the sections below.

Why can’t Google Analytics or other script-based analytics see AI crawlers?

Crawlers fetch the HTML and leave, so a tracker that runs in the browser never fires. Tools also drop known bots on purpose: Google says traffic from known bots is excluded automatically in Analytics and that you “cannot disable known bot traffic exclusion or see how much known bot traffic was excluded” (Google). AI crawler user agents has more on this. Tracking therefore has to happen at the server, the CDN or the log.

How do you track AI crawlers from an access log?

Run a script over the log file. Any server that writes the standard “combined” format, which is nginx’s default, already holds every crawler request with its address, path, status and user agent. This script counts each bot, separates robots.txt from page reads, counts errors, and checks each address against the list the operator publishes.

import collections, ipaddress, json, re, sys, urllib.request

OPENAI, CLAUDE = "https://openai.com/", "https://claude.com/crawling/bots.json"
PERPLEXITY = "https://www.perplexity.ai/"
GOOGLE = "https://developers.google.com/static/crawling/ipranges/common-crawlers.json"

BOTS = {  # user agent token: (kind, the operator's published IP list, if it has one)
    "GPTBot": ("training", OPENAI + "gptbot.json"),
    "ClaudeBot": ("training", CLAUDE),
    "CCBot": ("training", "https://index.commoncrawl.org/ccbot.json"),
    "meta-externalagent": ("training", None),
    "Amazonbot": ("training", None),
    "Bytespider": ("training", None),
    "OAI-SearchBot": ("search", OPENAI + "searchbot.json"),
    "Claude-SearchBot": ("search", CLAUDE),
    "PerplexityBot": ("search", PERPLEXITY + "perplexitybot.json"),
    "ChatGPT-User": ("user fetch", OPENAI + "chatgpt-user.json"),
    "Claude-User": ("user fetch", CLAUDE),
    "Perplexity-User": ("user fetch", PERPLEXITY + "perplexity-user.json"),
    "DuckAssistBot": ("user fetch", "https://duckduckgo.com/duckassistbot.json"),
    "Googlebot": ("search engine", GOOGLE),
    "Applebot": ("search engine", "https://search.developer.apple.com/applebot.json"),
    "bingbot": ("search engine", "https://www.bing.com/toolbox/bingbot.json"),
}
NAME = {token.lower(): token for token in BOTS}
TOKEN = re.compile("|".join(map(re.escape, BOTS)), re.I)
LINE = re.compile(r'(\S+) \S+ \S+ \[[^]]+\] "\S+ (\S+)[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')


def load(url, cache={}):
    """The operator's networks, or None if the list could not be read."""
    if url not in cache:
        try:
            request = urllib.request.Request(url, headers={"User-Agent": "ip-check"})
            prefixes = json.load(urllib.request.urlopen(request, timeout=20))["prefixes"]
            cache[url] = [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"]) for p in prefixes]
        except (OSError, ValueError, KeyError):
            cache[url] = None
    return cache[url]


def listed(ip, networks):
    try:
        address = ipaddress.ip_address(ip)
    except ValueError:
        return False
    return any(address in net for net in networks if net.version == address.version)


rows = collections.defaultdict(collections.Counter)
paths = collections.Counter()
focus = sys.argv[1].lower() if len(sys.argv) > 1 else None

for line in sys.stdin:
    match = LINE.match(line)
    found = match and TOKEN.search(match[4])
    if not found:
        continue
    ip, path, status, bot = match[1], match[2].split("?")[0], int(match[3]), NAME[found[0].lower()]
    row = rows[bot]
    row["reads"] += 1
    row["robots"] += path == "/robots.txt"
    row["errors"] += status >= 400
    url = BOTS[bot][1]
    networks = load(url) if url else None
    row["checked"] += networks is not None
    row["fake"] += networks is not None and not listed(ip, networks)
    if bot.lower() == focus:
        paths[path] += 1

if focus:
    for path, count in paths.most_common(20):
        print(f"{count:>7}  {path}")
    sys.exit()

print(f"{'bot':<20}{'kind':<15}{'reads':>7}{'robots':>8}{'4xx/5xx':>9}{'not in IP list':>16}")
for bot, row in sorted(rows.items(), key=lambda item: -item[1]["reads"]):
    fake = row["fake"] if row["checked"] else "no list"
    print(f"{bot:<20}{BOTS[bot][0]:<15}{row['reads']:>7}{row['robots']:>8}{row['errors']:>9}{fake:>16}")

zcat -f /var/log/nginx/access.log* | python3 ai_crawlers.py          # all bots
zcat -f /var/log/nginx/access.log* | python3 ai_crawlers.py gptbot   # one bot's top pages

We tested it on 8 November 2026 with Python 3 against a small log we wrote using the user agents the operators publish, real addresses taken from their lists, and one GPTBot line sent from a documentation-range address. It counted the fake:

bot                 kind             reads  robots  4xx/5xx  not in IP list
GPTBot              training             5       1        0               1
ClaudeBot           training             3       0        1               0
ChatGPT-User        user fetch           2       0        1               0
OAI-SearchBot       search               2       1        0               0
Claude-User         user fetch           1       0        0               0
PerplexityBot       search               1       0        0               0
meta-externalagent  training             1       0        0         no list
Googlebot           search engine        1       1        0               0

Reading the columns:

• robots is how many requests were for /robots.txt. A bot with nothing but robots.txt fetches has checked your rules and read little.
• 4xx/5xx points at the pages crawlers failed on.
• not in IP list counts requests whose address is not on the operator’s published list. The script reads each list fresh on every run, and when a list cannot be fetched it marks that bot “no list” instead of calling everything fake. “No list” also means the script has no JSON list for that bot: Amazon and ByteDance publish ranges on web pages the script does not parse, and Meta’s crawler page did not load when we checked on 8 November (on 3 November it published no list).

The limits are the ones logs always have. If a CDN or load balancer sits in front of your server, the address field holds its IP, and “not in IP list” means nothing until the real client address is logged. If the CDN answers from cache, that request never reaches the origin’s log. A host that gives you no access logs rules this option out. And a listed address is only as good as the list: operators add ranges, so a one-off miss is a reason to look closer, not proof of a fake.

How do you track AI crawlers with Cloudflare AI Crawl Control?

If your domain is proxied through Cloudflare, open AI Crawl Control in the dashboard; there is nothing to install. Cloudflare says it is available on all plans and works with “zero configuration”, and the only prerequisite is traffic flowing through its proxy.

What you get, per Cloudflare’s docs as of November 2026:

• Requests by crawler, operator, category, host and path, with status codes, data transfer and CSV export.
• A robots.txt violations table, showing crawlers that requested paths you disallowed. Cloudflare warns it compares past requests with your current rules, so a newly added Disallow flags requests that were fine at the time.
• Allow and block controls per crawler, enforced with a WAF custom rule. A custom block response needs a paid plan, and pay per crawl is in private beta.
• Referrals from AI platforms, on paid plans only.

Two limits matter. History is short: the get-started page says “on free plans, the Metrics tab only displays metrics for the past 24 hours”, and its plan table lists a “maximum 24-hour analytics window” under all plans, with configurable timeframes only for Enterprise with Bot Management. Identification is plan-bound too: “On the free plan, AI Crawl Control identifies AI crawlers based on their user agent strings”, and the stronger detection ID field needs Bot Management. So on a free plan, Cloudflare’s identification rests on the user agent, as invisit’s does. Cloudflare also exposes the same data through its GraphQL Analytics API, if you want to export it yourself.

How do you track AI crawlers with Vercel AI Traffic?

If you host on Vercel, open Observability, then AI Traffic. Vercel added the tab on 28 October 2026, and it counts AI bots requesting your pages, split into crawlers that index or train and assistants fetching for a user. That is a different measurement from people clicking through.

The tab shows the paths crawlers and assistants request most, paths where AI bots received an error, and paths requested as Markdown. Vercel’s changelog calls these “requests by verified AI bot”; its pages do not say how this tab decides a bot is verified. Observability itself is available on all plans, and none of the pages we read states a plan limit for this tab.

The limit is retention. Vercel’s limits table gives Observability’s data retention as 12 hours on Hobby, 1 day on Pro and 3 days on Enterprise, against 30 days with Observability Plus, and it does not list AI Traffic separately. Plus costs $1.20 per million events and is on by default for teams created or upgraded to Pro on or after 3 April 2026. Querying these metrics with vercel metrics from the CLI also needs Plus.

It only sees what runs on Vercel. To act on what you find, Vercel points to WAF custom rules, a separate product from this report.

How do you build a log pipeline for AI crawlers?

Send only the crawler lines into a database you query with SQL. This is the right call when you already run log infrastructure or need raw rows with IPs for months; if neither is true, the script above is enough.

This setup ran end to end in containers on 8 November 2026, with nginx, Vector 0.50 and ClickHouse 26.3. nginx writes a JSON line only for requests whose user agent matches, using the if= parameter of access_log, so ordinary traffic adds nothing to this file:

map $http_user_agent $ai_crawler {
  default 0;
  "~*GPTBot|ClaudeBot|OAI-SearchBot|ChatGPT-User|Claude-User|PerplexityBot" 1;
}

log_format crawler escape=json
  '{"time":"$time_iso8601","ip":"$remote_addr","host":"$host","method":"$request_method",'
  '"uri":"$request_uri","status":$status,"bytes":$body_bytes_sent,"ua":"$http_user_agent"}';

access_log /var/log/nginx/crawlers.jsonl crawler if=$ai_crawler;

The map and log_format go in the http block and the access_log line in your server block. An access_log in a server block replaces any log inherited from http, which we confirmed in a separate test, so repeat your normal access_log line there too.

Vector tails that file and writes to ClickHouse:

sources:
  nginx:
    type: file
    include: [/var/log/nginx/crawlers.jsonl]
    read_from: beginning
transforms:
  parsed:
    type: remap
    inputs: [nginx]
    source: |
      . = parse_json!(.message)
sinks:
  clickhouse:
    type: clickhouse
    inputs: [parsed]
    endpoint: http://clickhouse:8123
    table: crawler_hits
    date_time_best_effort: true
    skip_unknown_fields: true

CREATE TABLE crawler_hits (
  time DateTime,
  ip String,
  host LowCardinality(String),
  method LowCardinality(String),
  uri String,
  status UInt16,
  bytes UInt64,
  ua String
) ENGINE = MergeTree ORDER BY time;

SELECT
  extract(ua, '(?i)(GPTBot|ClaudeBot|OAI-SearchBot|ChatGPT-User|Claude-User|PerplexityBot)') AS bot,
  count() AS reads,
  countIf(path(uri) = '/robots.txt') AS robots,
  countIf(status >= 400) AS errors,
  uniqExact(path(uri)) AS pages
FROM crawler_hits
WHERE time > now() - INTERVAL 30 DAY
GROUP BY bot
ORDER BY reads DESC;

Hosted platforms can feed the same kind of table, though we checked these two only against their docs:

• Cloudflare Logpush is listed as available on all plans, subject to the datasets your account has. Its http_requests dataset carries ClientIP, ClientRequestPath and ClientRequestUserAgent, pushed to storage you choose. Each account includes 25 GB a month, then $0.03 per GB to R2 or $0.10 per GB to other destinations, as of November 2026. It “cannot backfill historical data”, so a failed job loses those logs.
• Vercel Log Drains are on Pro and Enterprise at $0.50 per GB, and their schema carries the user agent and client IP under proxy.

A pipeline gives you IP verification, as long as you keep the address and join it against the operators’ lists. It also gives you a system to maintain, which is the honest cost of this option.

How does invisit track AI crawlers, and where does it fall short?

invisit records crawler reads from something that runs server-side and shows them on a Crawlers page. @invisitdev/server calls it from your middleware in most server frameworks; a Cloudflare Worker routed in front of any origin does the same for static sites; and the dashboard’s Connect server panel also has a Node forwarder for nginx or Caddy access logs. We run the Worker version on invisit.dev, and what it recorded is public.

The page shows reads, AI share, assistant fetches and search share, a chart over time, and the bots, pages, categories and response codes behind them, with tabs for training, AI answers, search and other. Clicking a bot lists the pages it read, and clicking an error code lists the pages that returned it. Reads are kept with the rest of your analytics, with no retention window.

The limits, plainly:

• User agent only, no IP check. The collector classifies each read from the user agent and stores no address, so a fake GPTBot counts as GPTBot. Use the script above or the pipeline when that matters. It recognises 87 named crawlers, plus a catch-all that files any other user agent that calls itself a bot under Other.
• It needs a server, Worker or log forwarder. With none of those there is nothing to send the reads, and a page your CDN answers from cache never reaches middleware at your origin. If you cache HTML, put the recording at the edge.
• Status depends on placement. handle runs before your app and records every read as 200. Use proxy before and crawlers after to get the real status.
• It reports and does not act. There is no block control and no robots.txt violation report; Cloudflare has both.
• Fair use. Crawler reads are not billed, up to a line of three times the plan’s events: 60,000 reads a month on the free plan. Past it, reads stop being recorded until the next month, while visitor analytics carry on. A site with far more crawl than visitors can reach that, and a pipeline has no such line.
• Limited backfill. A forwarder can send lines at most 72 hours old, so history starts close to the day you connect.

What should you check once tracking is running?

Once reads are arriving, these checks turn them into decisions. Use the user agent lookup for anything you do not recognise, and the AI visibility tool to see how your site looks to these bots.

• Split reads by kind. Training, search and user fetches mean different things and call for different robots.txt rules.
• Compare robots.txt fetches with page reads for each bot, then check that crawlers you disallowed stopped reading pages after the operator’s stated delay.
• List the 4xx and 5xx pages for assistants. Each one is an answer that could not use your page.
• Check addresses for the bots you act on, using the operator’s list, before you trust a spike.
• Put assistant fetches next to visits from AI assistants. The first shows demand, the second shows clicks.
• Compare your setup with what other sites block, then decide with the robots.txt generator.

The trap to avoid is reading user agent counts as proof. Anyone can send GPTBot in a header, so count with whatever you have, and verify before you block or report a number you will be held to.