What is DuckDuckBot?
DuckDuckGo's web crawler, used to improve its search results.
Crawls pages for a classic search index. Blocking it removes you from that search engine.
DuckDuckGo says DuckDuckBot respects robots.txt but does not spell out a token; its user agent names it DuckDuckBot.
DuckDuckBot user agent
DuckDuckGo documents this user agent for DuckDuckBot. Version numbers in it change over time, so match on the name, not the whole string.
DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)Should you block DuckDuckBot?
Allow it unless you want out of that search engine. Blocking a major search crawler removes your pages from its results.
How to block DuckDuckBot in robots.txt
DuckDuckGo documents no robots.txt token for DuckDuckBot that we could confirm, so we don’t suggest one here. Block it at your server or firewall by user agent instead.
How to verify a request is really DuckDuckBot
Anyone can send any user agent, so the string alone proves nothing.
DuckDuckGo publishes the IP ranges DuckDuckBot crawls from. A request that claims to be DuckDuckBot from any other address is not DuckDuckBot.
See when DuckDuckBot reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against DuckDuckGo’s own documentation on . Something changed? Write to help@invisit.dev.