[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt)
# Content-Signal and __usage_ai-training explained
> What Content-Signal and __usage_ai-training are, who defined them, which AI companies read them and from when, and what to put in robots.txt today.
- Author: Miraç Berk
- Published: 2026-11-08
- Page: https://invisit.dev/blog/ai-training-preferences
- Tags: ai-traffic, crawlers, robots-txt
Content-Signal is a robots.txt line Cloudflare introduced in September 2025 to state preferences for search, AI input and AI training; as of 8 November 2026 we found no AI operator that says it reads it. `__usage_ai-training` is a robots.txt group name from an IETF draft dated 30 October 2026: Anthropic says it has begun recognizing it, Microsoft will honor it from 1 February 2027, Google intends to in early 2027, and OpenAI aims to in the coming months. In our scan of 10,870 readable robots.txt files on 8 November 2026, 762 sites (7.0%) carried a Content-Signal line and 22 (0.2%) used `__usage_ai-training`.
Our recommendation today: keep your rules for named crawlers, add a `__usage_ai-training` group, and treat Content-Signal as optional. Below are the exact syntax of each, a table of who has committed to what, how they interact with `Allow` and `Disallow`, one way a Content-Signal line can block your whole site, and files to copy.
## What is Content-Signal?
Content-Signal is a line inside a robots.txt group that says, with `yes` or `no`, how a crawler may use your content after fetching it. Cloudflare [announced it on 24 September 2025](https://blog.cloudflare.com/content-signals-policy/) and documents it at [contentsignals.org](https://contentsignals.org/). It defines three signals:
| Signal | Meaning, in Cloudflare's words |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `search` | "building a search index and providing search results (e.g., returning hyperlinks and short excerpts from your website's contents). Search does not include providing AI-generated search summaries." |
| `ai-input` | "inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers)" |
| `ai-train` | "training or fine-tuning AI models" |
```text title="robots.txt"
User-Agent: *
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /
```
Three details matter. Leaving a signal out does not mean no: per Cloudflare's policy text, the site "neither grants nor restricts permission via content signal" for that use. Cloudflare's site also shows a path form, `Content-Signal: /blog/ ai-train=no, search=yes, ai-input=no`, and groups addressed to a single bot. And Cloudflare's [robots.txt docs](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) say it is "testing" a fourth field, `use=`, taking `immediate`, `reference` or `full`.
Cloudflare says plainly that this is a statement of preference: "Some companies might simply ignore them." It also wraps the line in a comment block that calls any restriction an express reservation of rights under Article 4 of the EU copyright directive, while its own site warns that courts "may conclude that robots.txt files do not impose enforceable legal obligations". Cloudflare proposed the vocabulary at the IETF in [draft-romm-aipref-contentsignals](https://datatracker.ietf.org/doc/draft-romm-aipref-contentsignals/history/). That individual draft covers only the three labels and [expired on 4 April 2026](https://datatracker.ietf.org/doc/draft-romm-aipref-contentsignals/history/) with no new version.
## What is \_\_usage_ai-training?
`__usage_ai-training` is a policy token, a name you put on a normal `User-Agent` line to state a training preference instead of addressing a crawler. It comes from [draft-kelley-aipref-training-preference-signal-00](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-00), written by Kevin Kelley of Anthropic and dated 30 October 2026. It is an individual draft that the IETF AI Preferences working group [lists as related](https://datatracker.ietf.org/wg/aipref/documents/), not one of the group's own documents, and it expires on 3 May 2027 unless renewed.
```text title="robots.txt"
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
```
That is the draft's own example: a preference against AI training everywhere except `/press-kit/`. It adds no new grammar. The rules that follow from the draft:
- **A disallowed path is a preference against training, and an allowed path is a preference in favor.** Matching works as in RFC 9309, longest match first.
- **`User-Agent: *` never counts.** The draft says wildcard groups "MUST NOT be interpreted as matching the policy token".
- **No group means no preference**, so a robots.txt with no such line says nothing about training through this mechanism.
- **It does not control crawling.** "This policy token is not a directive for crawling behavior." A crawler may fetch the page and honor the preference afterwards.
- **It only covers generative models.** The definition comes from the working group's [vocabulary draft](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/): "Using an asset to modify the learned parameters of a generative AI model." Models used only for classification, ranking or scoring are outside it.
- **It applies as of the fetch.** The preferences that count are those "in effect when the asset was retrieved".
- **Private agreements override it**, in the draft's words: "Private agreements MAY override preferences expressed via this mechanism."
The token can share a group with crawler names: the draft says policy tokens and other product tokens "MAY be mixed in a group".
## Which AI companies read them, and from when?
Four operators have committed to `__usage_ai-training`, and only Anthropic describes support as started. We found no operator statement committing to Content-Signal. This is the state of each operator's own page on 8 November 2026:
| Operator | `__usage_ai-training` | Content-Signal |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| Anthropic | "[As of October 30, 2026, Anthropic has begun recognizing](https://support.claude.com/en/articles/8896518)" the signal, "rolling this out across all of our services over time". A rule naming ClaudeBot takes priority. | Page does not mention it |
| Microsoft | [From 1 February 2027](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/), as it crawls | Post does not mention it |
| Google | "[Intends to support](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)" it "in early 2027"; its [announcement](https://developers.google.com/search/blog/2026/10/ai-training-control) says "in 2027" | Not among the fields Google supports |
| OpenAI | "[We aim to implement the signal in the coming months](https://openai.com/index/helping-websites-express-ai-training-preferences/)". The post does not print the token or a date, and says existing crawler controls "will continue to apply and be respected". | Post does not mention it |
| Perplexity | [Crawler page](https://docs.perplexity.ai/guides/bots) silent. Perplexity says PerplexityBot is not used to crawl for foundation models. | Silent |
| Apple | [Applebot page](https://support.apple.com/en-us/119829) silent. Applebot-Extended remains the control. | Silent |
| Amazon | [Amazonbot page](https://developer.amazon.com/amazonbot) silent. Honors `noarchive` for model training. | Silent |
| Mistral | [Crawler page](https://docs.mistral.ai/robots) silent. MistralAI-Training remains the control. | Silent |
| Common Crawl | [CCBot page](https://commoncrawl.org/ccbot) silent | Silent |
Meta's crawler page did not load for us, so Meta is not in the table. Silence on these pages is what it looks like: no public commitment found, which is not proof of a decision either way.
On Content-Signal, the strongest evidence is Google's. Its [robots.txt documentation](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) says Google supports `user-agent`, `allow`, `disallow` and `sitemap`, and that it ignores invalid lines. Cloudflare's docs also warn that Search Console "may occasionally report `Syntax not understood`" for these lines, adding that Cloudflare has "observed no impact on crawling rates or SEO".
## How do these relate to Allow and Disallow rules?
`Allow` and `Disallow` say what a crawler may fetch; these newer signals say what it may do with a page it fetched. Both can sit in the same file and neither replaces the other. Four mechanisms are now in circulation:
| Mechanism | Defined by | Syntax | Status, November 2026 |
| ------------------------------ | ------------------------------------------------------------------------------------- | -------------------------------------------------- | ------------------------------------------------------------------ |
| `Disallow` for a named crawler | RFC 9309 | `User-agent: GPTBot` then `Disallow: /` | Internet standard; honored per each operator's docs |
| `__usage_ai-training` | Individual Internet-Draft | `User-Agent: __usage_ai-training` then `Disallow:` | Committed by four operators, staged rollout |
| `Content-Signal` | Cloudflare | `Content-Signal: ai-train=no` | No operator commitment found |
| `Content-Usage` | [AIPREF attachment draft](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/) | `Content-Usage: train-ai=n`, also an HTTP header | Working-group draft; its vocabulary "DO NOT REFLECT CONSENSUS" yet |
When rules disagree, the specific one wins where an operator has said so:
- **Anthropic:** "If your robots.txt file names ClaudeBot, we follow that rule, even if your general setting says something different."
- **Google:** if you opt out with the token "and Google-Extended is not explicitly specified", Google "plans to interpret that as if the publisher had disallowed Google-Extended". An explicit Google-Extended rule therefore overrides it.
- **Microsoft:** a page-level AI training preference "overrides the robots.txt setting".
- **OpenAI:** its post does not say which wins.
- **The draft:** "Specific expressions SHOULD take precedence over general ones."
Two caveats follow. Google-Extended covers Gemini training and grounding in Gemini Apps and Vertex AI together, so the token can also take you out of grounding: Google says sites that "specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps". Allowing it also allows Gemini training, since it is one token. And Microsoft says a usage `Allow` "does not override a robots.txt rule blocking Bingbot from crawling".
Microsoft goes further than the draft at page level, with a separate AI grounding preference that has no robots.txt form:
```html title="Page-level preferences for Bing, from 1 February 2027"
```
The same values work as an `X-Robots-Tag: Bingbot: ai-training:disallow` header, and `ai-grounding:disallowextended` limits grounding to Microsoft's own products. If you use `noarchive` or `nocache` today, Microsoft stops reading them for AI after 1 March 2027 and asks you to [move before then](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/).
## Can a Content-Signal line break my robots.txt?
Yes, in one specific layout. A Content-Signal line with no `Allow` or `Disallow` after it does not end a group, so a following `User-agent` line joins the same group and picks up its rules. This file looks like it blocks only GPTBot:
```text title="robots.txt (do not use)"
User-agent: *
Content-Signal: search=yes, ai-train=no
User-agent: GPTBot
Disallow: /
```
We ran it through [Google's open-source robots.txt parser](https://github.com/google/robotstxt), built from its repository on 8 November 2026, and it reported `DISALLOWED` for Googlebot and OAI-SearchBot as well as GPTBot. The `*` group has no rules of its own, so it merges with the next group. Python's `urllib.robotparser` and Protego allowed Googlebot and OAI-SearchBot on the same file, so parsers do not agree here. Google's [docs](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) state the general behavior: rules other than `allow`, `disallow` and `user-agent` "are ignored by the robots.txt parser", and it gives an example where two user agents end up sharing one `disallow`. RFC 9309 makes the same point about Sitemaps lines: such a record "MUST NOT terminate a group".
The fix is one line, and Cloudflare's own examples all include it:
```text title="robots.txt"
User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /
User-agent: GPTBot
Disallow: /
```
Google's parser reports Googlebot and OAI-SearchBot `ALLOWED` and GPTBot `DISALLOWED` on that version.
## How many sites use them?
762 of the 10,870 readable robots.txt files in our scan (7.0%) contained a Content-Signal line, and 22 (0.2%) used `__usage_ai-training`, nine days after the draft appeared. These come from [our study of the Tranco top 20,000](/blog/how-many-sites-block-ai-crawlers), fetched on 8 November 2026.
The most common Content-Signal value was `search=yes,ai-train=no,use=reference`, on 466 of the 762 sites. Those are the three signals Cloudflare's docs show for its managed robots.txt, though a robots.txt file alone cannot prove how a line got there.
Sites with the line block crawlers far more often: 546 of the 762 (71.7%) block GPTBot by name, against 1,101 of the other 10,108 (10.9%). Those 546 are 33.2% of all GPTBot blockers, which fits a line that tends to arrive alongside named rules rather than replacing them.
One snapshot, one fetch per site, from a list that includes infrastructure domains. The study counts what files say. It cannot show whether any crawler obeys them.
## What should I put in robots.txt today?
Keep your named rules, because they are the only part with a documented effect now, and add the shared token alongside them. It costs one group, parsers that do not know it ignore it, and Anthropic has started reading it. Add Content-Signal only if you want the policy statement on the record or your CDN manages it for you.
| Your goal | What to do |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| No training, stay in search and AI answers | File 1 |
| No training, except keep Gemini Apps grounding | File 2, and accept that Google-Extended also permits Gemini training |
| Allow training on some paths | File 3 |
| You are on Cloudflare | Set Training in AI Crawler and Agent access policies; Bot Preference Sync writes the managed lines |
| You use Bing `noarchive` or `nocache` for AI | Move to the new preferences before 1 March 2027 |
| Allow training everywhere | Nothing. No group means no preference |
```text title="File 1: no training, stay in search"
# Named crawlers and tokens: the rules that work today
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
# Shared training preference: read as support arrives
User-agent: __usage_ai-training
Disallow: /
# Everyone else, including search crawlers and user fetchers
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
```
```text title="File 2: keep Gemini Apps grounding"
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
User-agent: __usage_ai-training
Disallow: /
User-agent: Google-Extended
Allow: /
User-agent: *
Allow: /
```
```text title="File 3: allow training on one path only"
User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
User-agent: *
Allow: /
```
Google's parser returned the expected result for each file against the crawlers it names and for the token. File 1 leaves out [Bytespider](/bots/bytespider), whose operator documents no robots.txt behavior, as we explain in [the crawler post](/blog/ai-crawlers), where the fuller list of tokens and their caveats also lives.
To confirm what you published, fetch the live file and look for both lines:
```sh title="Terminal"
curl -sS https://example.com/robots.txt | grep -iE '^(user-agent: *__usage|content-signal)'
```
Your logs cannot confirm the effect. `__usage_ai-training` does not change crawling, so a crawler that honors it can keep fetching your pages exactly as before, and the operators publish no report of what they excluded.
The [robots.txt generator](/tools/robots-txt-generator) from invisit, the tool we build, writes Content-Signal lines and blocks crawlers by name, always with a rule in the group, so it does not produce the broken layout above. It does not write a `__usage_ai-training` group yet, so add that by hand. The [AI visibility checker](/tools/ai-visibility) reports a Content-Signal line as an optional item that never affects your score, and it does not look for the token.
## Checklist for your robots.txt
- [ ] Named training rules are still there, including Google-Extended and Applebot-Extended.
- [ ] A `User-agent: __usage_ai-training` group is added, with `Disallow: /` if you opt out.
- [ ] You decided whether losing Gemini Apps grounding is acceptable, and set Google-Extended accordingly.
- [ ] Any Content-Signal line sits in a group that also has an `Allow` or `Disallow`.
- [ ] The file is published on every subdomain, since robots.txt applies per host.
- [ ] If you rely on Bing's `noarchive` or `nocache`, the new preferences are in your calendar before 1 March 2027.
- [ ] You have not removed anything on the assumption that a draft is live for everyone.