What is Diffbot?
It crawls the web to build a general search engine, so sites can be found and cited from the Diffbot Knowledge Graph and web search services. Diffbot says it does not crawl pages to train generative AI foundation models.
Crawls pages for a classic search index. Blocking it removes you from that search engine.
Diffbot obeys robots.txt by default but says it can override it where it has an agreement with the site. Customers of its Crawl and Extract APIs can set their own user agent and turn robots.txt off. Diffbot-User is used for requests made on behalf of a human user.
Should you block Diffbot?
Allow it unless you want out of that search engine. Blocking a major search crawler removes your pages from its results.
How to block Diffbot in robots.txt
Add a group for its tokens to robots.txt at the root of your site. Diffbot says it follows robots.txt only in part, so this may not stop every fetch.
A bot follows only the group that names it, so it ignores your User-agent: * rules once it has a group of its own. Repeat any paths you also want kept from it there.
User-agent: Diffbot
User-agent: Diffbot-User
Disallow: /How to verify a request is really Diffbot
Anyone can send any user agent, so the string alone proves nothing.
Diffbot publishes no IP ranges or DNS names for Diffbot that we could find, so there is no reliable way to tell a real Diffbot from a scraper using its user agent.
See when Diffbot reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against Diffbot’s own documentation on . Something changed? Write to help@invisit.dev.