[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt)

# Bytespider

> Bytespider is an AI training crawler run by ByteDance. ByteDance documents Bytespider as the crawler for Toutiao Search: it follows links, takes page snapshots and builds a search index.

- Operator: ByteDance
- Kind: AI training crawler
- robots.txt token: none documented
- Source: https://zhanzhang.toutiao.com/docs/intro/26899 (checked 2026-11-05)
- Page: https://invisit.dev/bots/bytespider

## What is Bytespider?

ByteDance documents Bytespider as the crawler for Toutiao Search: it follows links, takes page snapshots and builds a search index.

Collects pages to train language models. Blocking it keeps your pages out of future training data, and does not remove you from AI search answers.

ByteDance's Chinese-only Bytespider page does not mention robots.txt or AI training. It also lists ten /24 IP ranges inline rather than as a downloadable file.

- [ByteDance’s documentation](https://zhanzhang.toutiao.com/docs/intro/26899)

## Bytespider user agent

ByteDance documents these user agents for Bytespider. Version numbers in it change over time, so match on the name, not the whole string.

```
Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/7.0 Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
```

## Should you block Bytespider?

Block it if you don’t want your pages used to train models; nothing else changes. Allow it if you’d rather future models know your product, docs or writing.

## How to block Bytespider in robots.txt

ByteDance documents no robots.txt token for Bytespider that we could confirm, so we don’t suggest one here. Block it at your server or firewall by user agent instead.

Toutiao's webmaster platform has a crawl frequency setting that ByteDance says takes effect within a day.

- [Build a robots.txt for AI bots](https://invisit.dev/tools/robots-txt-generator)

## How to verify a request is really Bytespider

Anyone can send any user agent, so the string alone proves nothing.

Or check by reverse DNS: look up the hostname of the request’s IP, confirm it ends in `bytedance.com`, then look that hostname up again and confirm it points back to the same IP.

## See when Bytespider reads your site

Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.

- [Track crawlers with invisit](https://invisit.dev/docs/server/sdk#crawler-reads)

## Common questions

### What is Bytespider?

Bytespider is an AI training crawler run by ByteDance. ByteDance documents Bytespider as the crawler for Toutiao Search: it follows links, takes page snapshots and builds a search index.

### Is Bytespider safe to block?

Block it if you don’t want your pages used to train models; nothing else changes. Allow it if you’d rather future models know your product, docs or writing.

### How do I block Bytespider?

ByteDance documents no robots.txt token for it that we could confirm, so block it at your server or firewall by its user agent.