Content-Signal and __usage_ai-training explained

Content-Signal is a robots.txt line Cloudflare introduced in September 2025 to state preferences for search, AI input and AI training; as of 8 November 2026 we found no AI operator that says it reads it. __usage_ai-training is a robots.txt group name from an IETF draft dated 30 October 2026: Anthropic says it has begun recognizing it, Microsoft will honor it from 1 February 2027, Google intends to in early 2027, and OpenAI aims to in the coming months. In our scan of 10,870 readable robots.txt files on 8 November 2026, 762 sites (7.0%) carried a Content-Signal line and 22 (0.2%) used __usage_ai-training.

Our recommendation today: keep your rules for named crawlers, add a __usage_ai-training group, and treat Content-Signal as optional. Below are the exact syntax of each, a table of who has committed to what, how they interact with Allow and Disallow, one way a Content-Signal line can block your whole site, and files to copy.

What is Content-Signal?

Content-Signal is a line inside a robots.txt group that says, with yes or no, how a crawler may use your content after fetching it. Cloudflare announced it on 24 September 2025 and documents it at contentsignals.org. It defines three signals:

User-Agent: *
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /

Three details matter. Leaving a signal out does not mean no: per Cloudflare’s policy text, the site “neither grants nor restricts permission via content signal” for that use. Cloudflare’s site also shows a path form, Content-Signal: /blog/ ai-train=no, search=yes, ai-input=no, and groups addressed to a single bot. And Cloudflare’s robots.txt docs say it is “testing” a fourth field, use=, taking immediate, reference or full.

Cloudflare says plainly that this is a statement of preference: “Some companies might simply ignore them.” It also wraps the line in a comment block that calls any restriction an express reservation of rights under Article 4 of the EU copyright directive, while its own site warns that courts “may conclude that robots.txt files do not impose enforceable legal obligations”. Cloudflare proposed the vocabulary at the IETF in draft-romm-aipref-contentsignals. That individual draft covers only the three labels and expired on 4 April 2026 with no new version.

What is __usage_ai-training?

__usage_ai-training is a policy token, a name you put on a normal User-Agent line to state a training preference instead of addressing a crawler. It comes from draft-kelley-aipref-training-preference-signal-00, written by Kevin Kelley of Anthropic and dated 30 October 2026. It is an individual draft that the IETF AI Preferences working group lists as related, not one of the group’s own documents, and it expires on 3 May 2027 unless renewed.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That is the draft’s own example: a preference against AI training everywhere except /press-kit/. It adds no new grammar. The rules that follow from the draft:

• A disallowed path is a preference against training, and an allowed path is a preference in favor. Matching works as in RFC 9309, longest match first.
• User-Agent: * never counts. The draft says wildcard groups “MUST NOT be interpreted as matching the policy token”.
• No group means no preference, so a robots.txt with no such line says nothing about training through this mechanism.
• It does not control crawling. “This policy token is not a directive for crawling behavior.” A crawler may fetch the page and honor the preference afterwards.
• It only covers generative models. The definition comes from the working group’s vocabulary draft: “Using an asset to modify the learned parameters of a generative AI model.” Models used only for classification, ranking or scoring are outside it.
• It applies as of the fetch. The preferences that count are those “in effect when the asset was retrieved”.
• Private agreements override it, in the draft’s words: “Private agreements MAY override preferences expressed via this mechanism.”

The token can share a group with crawler names: the draft says policy tokens and other product tokens “MAY be mixed in a group”.

Which AI companies read them, and from when?

Four operators have committed to __usage_ai-training, and only Anthropic describes support as started. We found no operator statement committing to Content-Signal. This is the state of each operator’s own page on 8 November 2026:

Meta’s crawler page did not load for us, so Meta is not in the table. Silence on these pages is what it looks like: no public commitment found, which is not proof of a decision either way.

On Content-Signal, the strongest evidence is Google’s. Its robots.txt documentation says Google supports user-agent, allow, disallow and sitemap, and that it ignores invalid lines. Cloudflare’s docs also warn that Search Console “may occasionally report Syntax not understood” for these lines, adding that Cloudflare has “observed no impact on crawling rates or SEO”.

How do these relate to Allow and Disallow rules?

Allow and Disallow say what a crawler may fetch; these newer signals say what it may do with a page it fetched. Both can sit in the same file and neither replaces the other. Four mechanisms are now in circulation:

When rules disagree, the specific one wins where an operator has said so:

• Anthropic: “If your robots.txt file names ClaudeBot, we follow that rule, even if your general setting says something different.”
• Google: if you opt out with the token “and Google-Extended is not explicitly specified”, Google “plans to interpret that as if the publisher had disallowed Google-Extended”. An explicit Google-Extended rule therefore overrides it.
• Microsoft: a page-level AI training preference “overrides the robots.txt setting”.
• OpenAI: its post does not say which wins.
• The draft: “Specific expressions SHOULD take precedence over general ones.”

Two caveats follow. Google-Extended covers Gemini training and grounding in Gemini Apps and Vertex AI together, so the token can also take you out of grounding: Google says sites that “specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps”. Allowing it also allows Gemini training, since it is one token. And Microsoft says a usage Allow “does not override a robots.txt rule blocking Bingbot from crawling”.

Microsoft goes further than the draft at page level, with a separate AI grounding preference that has no robots.txt form:

<meta name="Bingbot" content="ai-training:disallow" />
<meta name="Bingbot" content="ai-grounding:disallow" />

The same values work as an X-Robots-Tag: Bingbot: ai-training:disallow header, and ai-grounding:disallowextended limits grounding to Microsoft’s own products. If you use noarchive or nocache today, Microsoft stops reading them for AI after 1 March 2027 and asks you to move before then.

Can a Content-Signal line break my robots.txt?

Yes, in one specific layout. A Content-Signal line with no Allow or Disallow after it does not end a group, so a following User-agent line joins the same group and picks up its rules. This file looks like it blocks only GPTBot:

User-agent: *
Content-Signal: search=yes, ai-train=no

User-agent: GPTBot
Disallow: /

We ran it through Google’s open-source robots.txt parser, built from its repository on 8 November 2026, and it reported DISALLOWED for Googlebot and OAI-SearchBot as well as GPTBot. The * group has no rules of its own, so it merges with the next group. Python’s urllib.robotparser and Protego allowed Googlebot and OAI-SearchBot on the same file, so parsers do not agree here. Google’s docs state the general behavior: rules other than allow, disallow and user-agent “are ignored by the robots.txt parser”, and it gives an example where two user agents end up sharing one disallow. RFC 9309 makes the same point about Sitemaps lines: such a record “MUST NOT terminate a group”.

The fix is one line, and Cloudflare’s own examples all include it:

User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

User-agent: GPTBot
Disallow: /

Google’s parser reports Googlebot and OAI-SearchBot ALLOWED and GPTBot DISALLOWED on that version.

How many sites use them?

762 of the 10,870 readable robots.txt files in our scan (7.0%) contained a Content-Signal line, and 22 (0.2%) used __usage_ai-training, nine days after the draft appeared. These come from our study of the Tranco top 20,000, fetched on 8 November 2026.

The most common Content-Signal value was search=yes,ai-train=no,use=reference, on 466 of the 762 sites. Those are the three signals Cloudflare’s docs show for its managed robots.txt, though a robots.txt file alone cannot prove how a line got there.

Sites with the line block crawlers far more often: 546 of the 762 (71.7%) block GPTBot by name, against 1,101 of the other 10,108 (10.9%). Those 546 are 33.2% of all GPTBot blockers, which fits a line that tends to arrive alongside named rules rather than replacing them.

One snapshot, one fetch per site, from a list that includes infrastructure domains. The study counts what files say. It cannot show whether any crawler obeys them.

What should I put in robots.txt today?

Keep your named rules, because they are the only part with a documented effect now, and add the shared token alongside them. It costs one group, parsers that do not know it ignore it, and Anthropic has started reading it. Add Content-Signal only if you want the policy statement on the record or your CDN manages it for you.

# Named crawlers and tokens: the rules that work today
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Shared training preference: read as support arrives
User-agent: __usage_ai-training
Disallow: /

# Everyone else, including search crawlers and user fetchers
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: __usage_ai-training
Disallow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /

User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

User-agent: *
Allow: /

Google’s parser returned the expected result for each file against the crawlers it names and for the token. File 1 leaves out Bytespider, whose operator documents no robots.txt behavior, as we explain in the crawler post, where the fuller list of tokens and their caveats also lives.

To confirm what you published, fetch the live file and look for both lines:

curl -sS https://example.com/robots.txt | grep -iE '^(user-agent: *__usage|content-signal)'

Your logs cannot confirm the effect. __usage_ai-training does not change crawling, so a crawler that honors it can keep fetching your pages exactly as before, and the operators publish no report of what they excluded.

The robots.txt generator from invisit, the tool we build, writes Content-Signal lines and blocks crawlers by name, always with a rule in the group, so it does not produce the broken layout above. It does not write a __usage_ai-training group yet, so add that by hand. The AI visibility checker reports a Content-Signal line as an optional item that never affects your score, and it does not look for the token.

Checklist for your robots.txt

• Named training rules are still there, including Google-Extended and Applebot-Extended.
• A User-agent: __usage_ai-training group is added, with Disallow: / if you opt out.
• You decided whether losing Gemini Apps grounding is acceptable, and set Google-Extended accordingly.
• Any Content-Signal line sits in a group that also has an Allow or Disallow.
• The file is published on every subdomain, since robots.txt applies per host.
• If you rely on Bing’s noarchive or nocache, the new preferences are in your calendar before 1 March 2027.
• You have not removed anything on the assumption that a draft is live for everyone.