What is Claude-SearchBot?
It crawls the web to improve the relevance and accuracy of search results for Claude users.
Indexes pages so an AI search product can cite and link them in answers. Blocking it usually means you stop showing up as a source there.
Anthropic says blocking Claude-SearchBot may reduce your site's visibility and accuracy in Claude's search results.
Should you block Claude-SearchBot?
Usually allow it. It is how an AI search product finds pages to cite and link, so blocking it trades away being a source in those answers.
How to block Claude-SearchBot in robots.txt
Add a group for its token to robots.txt at the root of your site.
A bot follows only the group that names it, so it ignores your User-agent: * rules once it has a group of its own. Repeat any paths you also want kept from it there.
User-agent: Claude-SearchBot
Disallow: /How to verify a request is really Claude-SearchBot
Anyone can send any user agent, so the string alone proves nothing.
Anthropic publishes the IP ranges Claude-SearchBot crawls from. A request that claims to be Claude-SearchBot from any other address is not Claude-SearchBot.
See when Claude-SearchBot reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against Anthropic’s own documentation on . Something changed? Write to help@invisit.dev.