[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt)

# AI robots.txt generator

> Pick which AI bots may read your site, from training crawlers to AI search and assistants, and get a robots.txt that does exactly that, written the way crawlers actually parse it.

- Run it: https://invisit.dev/tools/robots-txt-generator (free, no account)
- Last changed: 2026-11-05

## Why the order of groups matters

robots.txt is read in groups. Each group starts with one or more `User-agent` lines and lists its rules under them. A crawler looks for the group that names it; only if none does, it falls back to `User-agent: *`. It never combines the two.

That catches a lot of sites out. A file that keeps `/admin` from everyone and then adds a group to allow GPTBot has just let GPTBot into `/admin`, because GPTBot now reads only its own group. This generator avoids it: bots you allow are never named, so they share the rules you give everyone, and bots you block share one group that disallows the whole site.

## Training, search and assistants are different bots

Most AI companies now run separate bots for separate jobs:

- **Training crawlers** collect pages for future models. Blocking them is the usual “opt out of AI training”.
- **Search crawlers** index pages for AI search, the way Googlebot does for Google. Blocking them means you stop being cited as a source.
- **Assistant fetchers** load a page because someone asked about it right then. They behave more like a visit than a crawl.

So “block AI” is rarely what you want. The No AI training preset blocks the first group and keeps the other two, which is where people coming from AI answers actually come from.

## After you publish it

Upload the file to the root of your domain, so it answers at `/robots.txt`. Each subdomain needs its own. Google caches it for up to a day, so changes take a little while to land.

Then check it. The [AI visibility checker](https://invisit.dev/tools/ai-visibility) reads your live file and tells you, bot by bot, who may read which page, and the [bot directory](https://invisit.dev/bots) explains each one.

## Common questions

### Does blocking GPTBot keep me out of ChatGPT?

No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, and pages fetched while someone chats come from ChatGPT-User. Each has its own token, so you can block training and still be found and cited.

### Why aren’t the bots I allow listed in the file?

A crawler follows only the group that names it most specifically. If GPTBot had a group of its own, it would ignore everything under User-agent: *, your Disallow rules included. Leaving allowed bots unnamed means they take the same rules as everyone else.

### Do AI bots actually respect robots.txt?

The big operators say their crawlers do. User-triggered fetchers, like an assistant opening a page someone pasted, are often treated as the person’s own visit and may not check it. Each bot page here says what its operator states.

### What is Content-Signal?

A line proposed by Cloudflare that says what your content may be used for: search, AI answers (ai-input) and AI training (ai-train). It doesn’t block anything by itself; it states your terms for the bots and companies that choose to honor it.

### What are Google-Extended and Applebot-Extended?

Tokens, not crawlers. Googlebot and Applebot still crawl for search, but disallowing these tokens tells Google and Apple not to use what they fetched for training their models.