invisit
DocsPricingStart free
Free tools

AI robots.txt generator

Pick which AI bots may read your site, from training crawlers to AI search and assistants, and get a robots.txt that does exactly that, written the way crawlers actually parse it.

.mdllms.txt
Start from
Block AI training

Block it if you don’t want your pages used to train models; nothing else changes. Allow it if you’d rather future models know your product, docs or writing.

Block AI search

Usually allow it. It is how an AI search product finds pages to cite and link, so blocking it trades away being a source in those answers.

Block AI assistants

Usually allow it. Each fetch is a person asking about your page right now, which is close to a visit. Blocking it in robots.txt may not stop it, since operators often treat these fetches as the person’s own.

Block SEO tools

Block it freely if the load bothers you. It sends no visitors; the only cost is that the tool’s users, your competitors included, see less of your site.

Content Signals

A newer line that says what your content may be used for, read by bots that honor it. Leave a signal unset to say nothing.

robots.txt

User-agent: *
Allow: /

Put it at the root of your site, as /robots.txt. Then run the AI visibility checker to see what each bot is allowed.

Why the order of groups matters

robots.txt is read in groups. Each group starts with one or more User-agent lines and lists its rules under them. A crawler looks for the group that names it; only if none does, it falls back to User-agent: *. It never combines the two.

That catches a lot of sites out. A file that keeps /admin from everyone and then adds a group to allow GPTBot has just let GPTBot into /admin, because GPTBot now reads only its own group. This generator avoids it: bots you allow are never named, so they share the rules you give everyone, and bots you block share one group that disallows the whole site.

Training, search and assistants are different bots

Most AI companies now run separate bots for separate jobs:

  • Training crawlers collect pages for future models. Blocking them is the usual “opt out of AI training”.
  • Search crawlers index pages for AI search, the way Googlebot does for Google. Blocking them means you stop being cited as a source.
  • Assistant fetchers load a page because someone asked about it right then. They behave more like a visit than a crawl.

So “block AI” is rarely what you want. The No AI training preset blocks the first group and keeps the other two, which is where people coming from AI answers actually come from.

After you publish it

Upload the file to the root of your domain, so it answers at /robots.txt. Each subdomain needs its own. Google caches it for up to a day, so changes take a little while to land.

Then check it. The AI visibility checker reads your live file and tells you, bot by bot, who may read which page, and the bot directory explains each one.

Common questions

Does blocking GPTBot keep me out of ChatGPT?

No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, and pages fetched while someone chats come from ChatGPT-User. Each has its own token, so you can block training and still be found and cited.

Why aren’t the bots I allow listed in the file?

A crawler follows only the group that names it most specifically. If GPTBot had a group of its own, it would ignore everything under User-agent: *, your Disallow rules included. Leaving allowed bots unnamed means they take the same rules as everyone else.

Do AI bots actually respect robots.txt?

The big operators say their crawlers do. User-triggered fetchers, like an assistant opening a page someone pasted, are often treated as the person’s own visit and may not check it. Each bot page here says what its operator states.

What is Content-Signal?

A line proposed by Cloudflare that says what your content may be used for: search, AI answers (ai-input) and AI training (ai-train). It doesn’t block anything by itself; it states your terms for the bots and companies that choose to honor it.

What are Google-Extended and Applebot-Extended?

Tokens, not crawlers. Googlebot and Applebot still crawl for search, but disallowing these tokens tells Google and Apple not to use what they fetched for training their models.

More free tools