What is Baiduspider?
Baidu's search engine crawler, which visits pages to build Baidu's search index.
Crawls pages for a classic search index. Blocking it removes you from that search engine.
Blocking Baiduspider blocks all Baidu user agents. Baidu says Baiduspider-cpro and Baiduspider-ads work under customer agreements and do not follow robots.txt, and Baiduspider-video does not support robots rules.
Baiduspider user agent
Baidu documents these user agents for Baiduspider. Version numbers in it change over time, so match on the name, not the whole string.
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko)Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0;+http://www.baidu.com/search/spider.html)Should you block Baiduspider?
Allow it unless you want out of that search engine. Blocking a major search crawler removes your pages from its results.
How to block Baiduspider in robots.txt
Add a group for its tokens to robots.txt at the root of your site.
A bot follows only the group that names it, so it ignores your User-agent: * rules once it has a group of its own. Repeat any paths you also want kept from it there.
User-agent: Baiduspider
User-agent: Baiduspider-image
User-agent: Baiduspider-video
User-agent: Baiduspider-news
Disallow: /How to verify a request is really Baiduspider
Anyone can send any user agent, so the string alone proves nothing.
Or check by reverse DNS: look up the hostname of the request’s IP, confirm it ends in baidu.com or baidu.jp, then look that hostname up again and confirm it points back to the same IP.
See when Baiduspider reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against Baidu’s own documentation on . Something changed? Write to help@invisit.dev.