What is Bytespider?
ByteDance documents Bytespider as the crawler for Toutiao Search: it follows links, takes page snapshots and builds a search index.
Collects pages to train language models. Blocking it keeps your pages out of future training data, and does not remove you from AI search answers.
ByteDance's Chinese-only Bytespider page does not mention robots.txt or AI training. It also lists ten /24 IP ranges inline rather than as a downloadable file.
Bytespider user agent
ByteDance documents these user agents for Bytespider. Version numbers in it change over time, so match on the name, not the whole string.
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/7.0 Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)Should you block Bytespider?
Block it if you don’t want your pages used to train models; nothing else changes. Allow it if you’d rather future models know your product, docs or writing.
How to block Bytespider in robots.txt
ByteDance documents no robots.txt token for Bytespider that we could confirm, so we don’t suggest one here. Block it at your server or firewall by user agent instead.
Toutiao's webmaster platform has a crawl frequency setting that ByteDance says takes effect within a day.
How to verify a request is really Bytespider
Anyone can send any user agent, so the string alone proves nothing.
Or check by reverse DNS: look up the hostname of the request’s IP, confirm it ends in bytedance.com, then look that hostname up again and confirm it points back to the same IP.
See when Bytespider reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against ByteDance’s own documentation on . Something changed? Write to help@invisit.dev.