How to verify GPTBot, ClaudeBot and other AI crawler IPs

To verify an AI crawler, check the request’s source IP against the list the operator publishes, or run a forward-confirmed reverse DNS lookup where the operator documents one. The user agent alone proves nothing. On 8 November 2026, 10 operators published 21 machine-readable IP lists, from OpenAI’s 18-prefix gptbot.json to Amazon’s 1,292 addresses for Amazonbot; Meta, Mistral’s training crawler, xAI and Cohere publish none.

Below: every list with its prefix count and date, a verifier in TypeScript and in shell that we ran against the live lists, how often to refresh, reverse DNS results for each operator that documents it, what Cloudflare’s verified bots and Web Bot Auth actually check, and what to do when a list answers 403. For what each bot does, see AI crawler user agents; to find them in your logs first, see how to detect AI crawlers.

Which AI crawlers publish their IP addresses?

Ten operators do, as of 8 November 2026. Counts and creationTime come from our own fetch of each file that day at 15:47 UTC; a dash means the file has no IPv6 entries.

Things worth knowing before you automate this:

• Amazon does not serve JSON files. Each of its three URLs returns an HTML page with the JSON inside it. The formats differ too: Amazonbot and Amzn-User list bare addresses under ipv4Prefix, and Amzn-SearchBot lists /32s under ip_prefix. A parser that only reads ipv4Prefix gets nothing from Amzn-SearchBot, and one that expects a / breaks on the bare addresses.
• DuckDuckGo’s two files are identical. duckassistbot.json and duckduckbot.json were byte-for-byte the same file on 8 November, so the list cannot tell you which of the two bots sent a request.
• creationTime is not a freshness date. Bing’s says 3 January 2024, 1,040 days before our fetch, and Bing’s own docs say “Refresh this list daily as we can change this list any time” (Bing). Treat it as the file’s own label and compare contents instead.
• Google’s fetcher list does not mean Google. Google says user-triggered-fetchers.json is for cases such as “a site running on Google Cloud” fetching a feed for its users, and those addresses resolve to gae.googleusercontent.com (Google). A hit there means a Google Cloud customer, not a Google crawler.
• Anthropic’s browser has its own file. Claude-Agent, the hosted browser, is published separately at claude.com/.well-known/sail (9 addresses, modified 2026-10-22, expires_after 3600). It is TOML in a draft format, and Anthropic’s article calls that format “not yet a standard”.

Which AI crawlers publish no IP list?

These operators document no list you can check against, as of 8 November 2026:

Two more publish addresses as text on a page rather than a file. ByteDance lists 10 /24 ranges for Bytespider. You.com says YouBot comes from 68.67.112.0/24.

Without a list, a request claiming to be one of these bots cannot be proved or disproved by address. Treat it as a claim and rely on your server’s own limits.

A verifier you can copy

This TypeScript module reads every list format above, including Amazon’s HTML pages, and refuses to answer when a fetch fails or a file has no prefixes. It runs on Bun, or on Node 24 with --experimental-strip-types, using only node:net and node:dns.

import { promises as dns } from 'node:dns'
import { BlockList, isIP } from 'node:net'

// Say who you are; some list hosts refuse anonymous scripts.
const USER_AGENT = 'crawler-ip-check/1.0 (+https://example.com/contact)'

export const LISTS = {
  gptbot: 'https://openai.com/gptbot.json',
  'oai-searchbot': 'https://openai.com/searchbot.json',
  'chatgpt-user': 'https://openai.com/chatgpt-user.json',
  'oai-adsbot': 'https://openai.com/adsbot.json',
  anthropic: 'https://claude.com/crawling/bots.json',
  perplexitybot: 'https://www.perplexity.ai/perplexitybot.json',
  'perplexity-user': 'https://www.perplexity.ai/perplexity-user.json',
  'mistralai-user': 'https://mistral.ai/mistralai-user-ips.json',
  'mistralai-index': 'https://mistral.ai/mistralai-index-ips.json',
  duckassistbot: 'https://duckduckgo.com/duckassistbot.json',
  applebot: 'https://search.developer.apple.com/applebot.json',
  bingbot: 'https://www.bing.com/toolbox/bingbot.json',
  ccbot: 'https://index.commoncrawl.org/ccbot.json',
  amazonbot: 'https://developer.amazon.com/amazonbot/ip-addresses/',
  'amzn-searchbot': 'https://developer.amazon.com/amazonbot/searchbot-ip-addresses/',
  'amzn-user': 'https://developer.amazon.com/amazonbot/live-ip-addresses/',
  'google-common': 'https://developers.google.com/static/crawling/ipranges/common-crawlers.json',
  'google-special': 'https://developers.google.com/static/crawling/ipranges/special-crawlers.json',
  'google-fetchers-google':
    'https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json',
  'google-fetchers-gcp':
    'https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json',
  'google-agents':
    'https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json',
} as const

export interface CrawlerList {
  readonly url: string
  readonly creationTime: string | null
  readonly prefixCount: number
  contains(ip: string): boolean
}

const familyOf = (ip: string): 'ipv4' | 'ipv6' => (isIP(ip) === 6 ? 'ipv6' : 'ipv4')

/* Reads every operator's format: Amazon wraps JSON in an HTML page, uses bare addresses or
   /32s, and names the key ip_prefix on one of its three lists. */
export async function loadList(url: string): Promise<CrawlerList> {
  const response = await fetch(url, { headers: { 'User-Agent': USER_AGENT } })
  if (!response.ok) throw new Error(`${url} answered ${response.status}`)
  const body = await response.text()

  const prefixes = [...body.matchAll(/"(?:ipv4Prefix|ipv6Prefix|ip_prefix)"\s*:\s*"([^"]+)"/g)].map(
    (match) => match[1] as string,
  )
  if (prefixes.length === 0) throw new Error(`${url} contained no prefixes`)

  const blockList = new BlockList()
  for (const prefix of prefixes) {
    const [address, bits] = prefix.split('/') as [string, string | undefined]
    if (isIP(address) === 0) throw new Error(`${url} has an invalid prefix: ${prefix}`)
    if (bits === undefined) blockList.addAddress(address, familyOf(address))
    else blockList.addSubnet(address, Number(bits), familyOf(address))
  }

  return {
    url,
    creationTime: body.match(/"creationTime"\s*:\s*"([^"]+)"/)?.[1] ?? null,
    prefixCount: prefixes.length,
    contains: (ip) => blockList.check(ip, familyOf(ip)),
  }
}

/* Reverse lookup, suffix check, then forward lookup back to the same address. */
export async function verifyReverseDns(ip: string, suffixes: readonly string[]): Promise<boolean> {
  const hostnames = await dns.reverse(ip).catch((error: NodeJS.ErrnoException) => {
    if (error.code === 'ENOTFOUND' || error.code === 'ENODATA') return [] as string[]
    throw error
  })
  const hostname = hostnames.find((name) =>
    suffixes.some((suffix) => name.toLowerCase().endsWith(`.${suffix}`)),
  )
  if (!hostname) return false

  const family = familyOf(ip)
  const addresses = await (family === 'ipv6' ? dns.resolve6(hostname) : dns.resolve4(hostname))
  const expected = new BlockList()
  expected.addAddress(ip, family)
  return addresses.some((address) => expected.check(address, family))
}

if (import.meta.main) {
  const [name, ip] = process.argv.slice(2) as [keyof typeof LISTS, string]
  const url = LISTS[name]
  if (!url || !ip || isIP(ip) === 0) {
    console.error('usage: verify-crawler-ip <list-name> <ip>')
    process.exit(2)
  }
  const list = await loadList(url)
  const listed = list.contains(ip)
  console.log(
    `${ip} ${listed ? 'listed' : 'not listed'} in ${name} (${list.prefixCount} prefixes, created ${list.creationTime ?? 'unknown'})`,
  )
  process.exit(listed ? 0 : 1)
}

bun verify-crawler-ip.ts gptbot 132.196.86.10
# 132.196.86.10 listed in gptbot (18 prefixes, created 2026-09-22T02:00:07.000000)
bun verify-crawler-ip.ts gptbot 8.8.8.8
# 8.8.8.8 not listed in gptbot (18 prefixes, created 2026-09-22T02:00:07.000000)

The same check in shell, with curl, grep and awk only. It handles IPv4 and IPv6, and exits 0 for listed, 1 for not listed and 2 when the list cannot be read:

#!/usr/bin/env bash
# usage: verify-crawler-ip.sh <list-url> <ip>
# exit codes: 0 listed, 1 not listed, 2 the list could not be read
set -uo pipefail

url=$1
ip=${2#::ffff:}   # an IPv4 client seen by a dual-stack server

prefixes=$(curl -fsSL --max-time 30 -A 'crawler-ip-check/1.0 (+https://example.com/contact)' "$url") || {
  echo "could not fetch $url" >&2; exit 2; }

printf '%s' "$prefixes" |
  grep -oE '"(ipv4Prefix|ipv6Prefix|ip_prefix)"[[:space:]]*:[[:space:]]*"[^"]+"' |
  cut -d'"' -f4 |
  awk -v ip="$ip" '
    function binary(value, width,    out) {
      out = ""
      while (width-- > 0) { out = (value % 2) out; value = int(value / 2) }
      return out
    }
    function hexdigit(c) { return index("0123456789abcdef", tolower(c)) - 1 }
    function group(g,    i, out) {
      while (length(g) < 4) g = "0" g
      out = ""
      for (i = 1; i <= 4; i++) out = out binary(hexdigit(substr(g, i, 1)), 4)
      return out
    }
    function v6(addr,    pos, left, right, a, b, na, nb, i, out, missing) {
      pos = index(addr, "::")
      if (pos) { left = substr(addr, 1, pos - 1); right = substr(addr, pos + 2) }
      else { left = addr; right = "" }
      na = left == "" ? 0 : split(left, a, ":")
      nb = right == "" ? 0 : split(right, b, ":")
      missing = pos ? 8 - na - nb : 0
      out = ""
      for (i = 1; i <= na; i++) out = out group(a[i])
      for (i = 1; i <= missing; i++) out = out group("0")
      for (i = 1; i <= nb; i++) out = out group(b[i])
      return out
    }
    function v4(addr,    a, out, i) {
      split(addr, a, ".")
      out = ""
      for (i = 1; i <= 4; i++) out = out binary(a[i] + 0, 8)
      return out
    }
    function bits(addr) { return index(addr, ":") ? v6(addr) : v4(addr) }
    BEGIN { target = bits(ip); count = 0; found = 0 }
    {
      count++
      n = split($0, parts, "/")
      addr = parts[1]
      pb = bits(addr)
      len = n > 1 ? parts[2] + 0 : length(pb)
      if (length(pb) == length(target) && substr(pb, 1, len) == substr(target, 1, len)) {
        found = 1; hit = $0
      }
    }
    END {
      if (count == 0) { print "no prefixes in list" > "/dev/stderr"; exit 2 }
      if (found) { print "listed in " hit; exit 0 }
      print "not listed (" count " prefixes checked)"; exit 1
    }
  '
exit "${PIPESTATUS[3]}"

./verify-crawler-ip.sh https://claude.com/crawling/bots.json 216.73.216.1
# listed in 216.73.216.0/22
./verify-crawler-ip.sh https://claude.com/crawling/bots.json 1.1.1.1
# not listed (14 prefixes checked)

What we ran on 8 November 2026 against the live lists, in both implementations:

We also generated 5,500 cases across the downloaded lists, 150 addresses inside each list’s prefixes and 100 random addresses per list, and compared both verifiers with Python’s ipaddress module as the reference. The TypeScript version checked all 5,500 and the shell version the first 40 cases per list, with zero mismatches.

The module also has verifyReverseDns, which needs the suffixes from the next section. Remember to pass the real client address: behind a proxy that is the last x-forwarded-for entry your own proxy added, as the detection post explains.

How often should I refresh the lists?

Once a day. Of the pages we read, only Bing names a cadence (“daily”). Perplexity says only to refresh “periodically” and that the addresses are “updated regularly”, and OpenAI, Anthropic, Google and Apple state no interval on the pages we read. Anthropic’s Claude-Agent file carries an expires_after of one hour. The lists do move: chatgpt-user.json went from the 235 prefixes the pillar post recorded on 3 November to 237 five days later.

Keep the last good copy when a fetch fails. Replacing a list with an empty file turns every real crawler into a fake, and a daily job will fail sooner or later.

#!/bin/sh
# Refreshes every list into a directory; a failed fetch keeps the previous copy and exits non-zero.
dir=${1:-/var/lib/crawler-ips}
mkdir -p "$dir"
status=0

while read -r name url; do
  [ -z "$name" ] && continue
  tmp=$(mktemp "$dir/.$name.XXXXXX")
  if curl -fsSL --max-time 30 -A 'crawler-ip-check/1.0 (+https://example.com/contact)' "$url" -o "$tmp" \
    && grep -qE '"(ipv4Prefix|ipv6Prefix|ip_prefix)"' "$tmp"; then
    mv "$tmp" "$dir/$name"
  else
    echo "kept previous $name, refresh failed: $url" >&2
    rm -f "$tmp"
    status=1
  fi
done <<'LISTS'
gptbot https://openai.com/gptbot.json
oai-searchbot https://openai.com/searchbot.json
chatgpt-user https://openai.com/chatgpt-user.json
anthropic https://claude.com/crawling/bots.json
perplexitybot https://www.perplexity.ai/perplexitybot.json
perplexity-user https://www.perplexity.ai/perplexity-user.json
applebot https://search.developer.apple.com/applebot.json
google-common https://developers.google.com/static/crawling/ipranges/common-crawlers.json
amzn-searchbot https://developer.amazon.com/amazonbot/searchbot-ip-addresses/
LISTS
exit "$status"

Run it once a day from cron or a systemd timer, and alert on a non-zero exit. We ran this script against the live URLs (all 9 files written), then against a local server that answered 403 and one that answered an HTML page with no prefixes: both times it kept the previous file, exited 1, and said which list failed. Extend the bottom list to the operators you care about from the table above.

How do I verify a crawler with reverse DNS?

Look up the name for the IP, check that it ends in the operator’s domain, then look the name up and confirm it returns the same IP. Skip any of those steps and you are trusting a record the owner of the address block controls. The pillar post lists the suffixes for Google, Apple, Microsoft, Common Crawl and ByteDance; since then we also read You.com’s and Yandex’s, and Google’s page now adds geo-crawl-***.geo.googlebot.com hostnames.

#!/usr/bin/env bash
# usage: verify-reverse-dns.sh <ip> <suffix> [suffix...]
# exit codes: 0 verified, 1 not verified
ip=$1; shift

hostname=$(dig +short -x "$ip" | head -n 1 | sed 's/\.$//')
if [ -z "$hostname" ]; then echo "no reverse DNS record for $ip"; exit 1; fi

for suffix in "$@"; do
  case "$hostname" in
    *."$suffix")
      if { dig +short A "$hostname"; dig +short AAAA "$hostname"; } | grep -qxF "$ip"; then
        echo "verified: $hostname"
        exit 0
      fi
      echo "forward lookup for $hostname does not return $ip"
      exit 1
      ;;
  esac
done
echo "$hostname is not under an expected domain"
exit 1

case with *."$suffix" matches only a real subdomain, so evilgooglebot.com fails. Results from 8 November 2026:

Two caveats from running it. Common Crawl says reverse DNS is not yet set up over IPv6. ByteDance’s page only describes the reverse step, and its own example address 111.225.148.250 had no reverse record for us; of nine addresses we sampled across its ranges, five returned a crawl.bytedance.com name and four returned none, and the one we forward-checked, bytespider-110-249-202-10.crawl.bytedance.com, resolved to nothing. Forward confirmation is the point of the check, so a Bytespider result that passes only the reverse half is weak evidence. Yandex documents the full forward and reverse check against yandex.ru, yandex.net and yandex.com (Yandex).

How does Cloudflare verify bots?

Cloudflare keeps its own verified list and exposes the result as request fields. A verified bot must, in its words, declare itself through “a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS” and show “non-abusive behavior”, including obeying robots.txt. Cloudflare removes a service from the list on a breach, and its example breaches include “an AI Crawler that does not respect the crawl-delay directive in robots.txt”. We read these docs on 8 November 2026 and did not run the expressions below against a live zone.

Use this to catch fakes in a WAF rule. cf.client.bot is true for known good bots and, unlike the cf.bot_management.* fields, carries no Enterprise add-on note in the fields reference. contains is case-sensitive, so name the casing you expect:

(http.user_agent contains "GPTBot" and not cf.client.bot)

cf.verified_bot_category gives a category such as AI Crawler, AI Search or AI Assistant. Cloudflare says that since 1 July 2026 there is “no longer a meaningful distinction” between AI Search and traditional search and that the old value stays for backward compatibility, so do not build new logic on AI Search. The list is Cloudflare’s: a bot that is honest but unlisted will not be marked. Cloudflare’s examples for the AI Crawler category include “ChatGPT bot”, and it shares the directory through Cloudflare Radar; check there that the bot you care about is in it before you block on this field.

Perplexity warns about one consequence for allow rules: “Cloudflare’s Bot Fight Mode can’t be skipped with WAF custom rules” (Perplexity). Vercel’s firewall takes the same approach, with a verified bots directory checked by IP, reverse DNS and Web Bot Auth.

What is Web Bot Auth and who uses it?

Web Bot Auth is a way for a bot to sign requests with a key, so the request proves who sent it regardless of IP address. It is built on RFC 9421, HTTP Message Signatures. The agent part is still an IETF working group draft, at revision 01 on 29 October 2026, so expect changes.

A signed request carries Signature, Signature-Input and Signature-Agent headers. You fetch the signer’s keys from its /.well-known/http-message-signatures-directory and verify. As of 8 November 2026:

We fetched all four key directories on 8 November and confirmed they serve Ed25519 keys, but we did not verify a live signed request, because none reached a server we control.

Two practical points. First, the absence of a signature proves nothing: Google tells you to “fall back to the established methods” because not every request is signed. Second, the header comes in two shapes. OpenAI sends a quoted string; Google sends a labelled dictionary. Cloudflare’s docs say its verifier fails a message that “uses the dictionary form from later drafts”, so check which form your verifier accepts before you rely on it. OpenAI’s verification steps, for any CDN or server without built-in support, are to confirm Signature-Agent is exactly "https://chatgpt.com" with the quotes, fetch the key, verify against RFC 9421, and allow only on success.

On Cloudflare, signature-verified agents are now part of Verified bots; the Enterprise field cf.bot_management.signed_agent marks them (fields reference).

What should I do when an IP list answers 403?

Treat it as a failed fetch and keep your previous copy. Do not change your request to look like a browser.

This matters most for lists behind a bot-protection layer. Perplexity’s two files are served through Cloudflare (server: cloudflare, cf-cache-status: HIT), and an earlier scripted curl during research for this post got a 403. When we re-tested from one connection on 8 November, all 24 requests, three rounds of both files with curl’s default user agent, Python’s urllib user agent, our named research user agent and no user agent, returned 200, and so did Node’s fetch and Python’s urllib. Whether a list answers can depend on where you are asking from and when, and that is out of your hands.

Both verifiers above treat a 403 as an error rather than an empty list: the shell script exits 2 and the TypeScript module throws, which we confirmed against a local server returning 403.

To fetch them honestly:

• Send a real User-Agent that names you and gives a contact, as the scripts here do.
• Back off and retry later, and refresh daily so one bad run does not matter.
• If your server’s network is refused and a browser works, save the file by hand from a browser and load it from disk. That is one person reading a public page.
• Ask the operator. Perplexity’s crawler page gives no contact for this; Anthropic gives claudebot@anthropic.com for crawler questions.

Do not fake a browser user agent or rotate through proxy networks to get past the block. That is the behaviour the block exists to stop, and these pages publish the lists so that site owners can fetch them openly.

What does a match actually prove?

It proves the request came from an address the operator says it uses. It does not prove intent or which product sent it.

• Anthropic’s one list covers ClaudeBot, Claude-SearchBot and Claude-User. A match tells you the operator, and the user agent then tells you which bot, so check both.
• Perplexity recommends the same pairing: “combine both User-Agent string matching and IP address verification”. Its rule example matches each bot’s own user agent against that bot’s own list.
• Blocking by IP is not an opt-out. Anthropic warns that this “may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file”. Use robots.txt for preferences and these lists for checking.

invisit, the tool we build, cannot do this for you: its crawler classification works on the user agent alone, so a spoofed GPTBot counts as GPTBot. Each /bots page links the operator’s list and documented DNS suffix for one-off checks; the scripts here are what to use in production.

Checklist

• Log the real client IP, not your load balancer’s.
• Refresh the lists daily, keep the last good copy, and alert on failure.
• Parse every format: Amazon’s HTML pages, bare addresses, ip_prefix, IPv6.
• Match the IP against the list for the bot the user agent names, not any list.
• Do not treat user-triggered-fetchers.json as “Google the crawler”.
• Use forward-confirmed reverse DNS where documented, and accept that IPv6 and Bytespider have gaps.
• Treat a missing signature as no information, and a valid one as proof of the signer’s key.
• Never fake a browser to fetch a list.