What is CCBot?
It crawls the web for Common Crawl, a non-profit that publishes an open repository of web crawl data that anyone can use.
Collects pages to train language models. Blocking it keeps your pages out of future training data, and does not remove you from AI search answers.
Common Crawl says other crawlers falsely claim to be CCBot. Reverse DNS is not yet set up for its IPv6 addresses.
CCBot user agent
Common Crawl documents this user agent for CCBot. Version numbers in it change over time, so match on the name, not the whole string.
CCBot/2.0 (https://commoncrawl.org/faq/)Should you block CCBot?
Block it if you don’t want your pages used to train models; nothing else changes. Allow it if you’d rather future models know your product, docs or writing.
How to block CCBot in robots.txt
Add a group for its token to robots.txt at the root of your site.
A bot follows only the group that names it, so it ignores your User-agent: * rules once it has a group of its own. Repeat any paths you also want kept from it there.
User-agent: CCBot
Disallow: /How to verify a request is really CCBot
Anyone can send any user agent, so the string alone proves nothing.
Common Crawl publishes the IP ranges CCBot crawls from. A request that claims to be CCBot from any other address is not CCBot.
Or check by reverse DNS: look up the hostname of the request’s IP, confirm it ends in crawl.commoncrawl.org, then look that hostname up again and confirm it points back to the same IP.
See when CCBot reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against Common Crawl’s own documentation on . Something changed? Write to help@invisit.dev.