Do websites actually use llms.txt? We checked 20,000

Some do, but a minority. On 8 November 2026 we requested /llms.txt from the Tranco top 20,000 domains: 2,053 of the 14,671 that answered (14.0%) serve a real file, rising to 130 of 699 (18.6%) in the top 1,000. Publishing is the easy half. We found no statement from OpenAI, Anthropic, Google or Perplexity that their crawlers or assistants read llms.txt files on other people’s sites, and Google says outright that Search ignores them.

Below: the adoption numbers and their limits, what the llmstxt.org proposal asks for, what each operator’s own pages say, what one small site saw in its logs, a decision table for whether to bother, and a working example file.

How many websites have an llms.txt file?

14.0% of the sites we could reach have one at the root of their domain. A site counts when /llms.txt answers HTTP 200 with a non-empty body that is not an HTML page.

The top 1,000 is the only band that stands out. Between ranks 1,001 and 10,000 the rate is 904 of 6,608 (13.7%), and between 10,001 and 20,000 it is 1,019 of 7,364 (13.8%). Below the very top, popularity makes no visible difference.

Most of the files look like what the proposal describes. 1,902 of the 2,053 (92.6%) open with a # heading as the format asks. The median file is 7,132 bytes, the middle half run from about 3.6 KB to 15 KB, and 49 are under 100 bytes.

A raw status check overstates this badly. 3,651 reachable sites answered 200 to /llms.txt, which would read as 24.9% adoption, about 1.8 times the real share. 1,598 of those responses were HTML or empty: a 200 is not a file. Any survey that counts status codes alone inherits that error.

Limits of the count, so you can discount it properly:

• One path. We checked the root only. The proposal also allows files at a subpath such as /docs/llms.txt, so documentation that lives under a path is missed.
• A list of domains, not websites. Tranco ranks infrastructure such as CDNs and ad servers too. 5,329 of the 20,000 did not answer at all, and they are out of the denominator.
• Publishing is not always a decision. The proposal lists platforms that generate the file for you, including Mintlify, GitBook, Yoast SEO and Wix, which it says generates one “for every Wix site”. We cannot separate those from files someone wrote by hand.
• One snapshot. The per-site data is public, and the method is in our study of who blocks AI crawlers.

What does the llms.txt proposal actually say?

llms.txt is a proposal by Jeremy Howard for a Markdown file that gives AI agents a short, curated guide to a site. It is an informal convention with no standards body behind it: the proposal page, dated 3 September 2024 and revised as v2 on 10 August 2026, calls itself “this informal overview”, and it is developed in a public GitHub repository.

The format is small. In order, a file contains:

1. An H1 with the name of the site. This is the only required part.
2. A blockquote with a short summary.
3. Optional paragraphs or lists with any detail an agent should know, with no headings.
4. Optional H2 sections, each a list of [name](url) links with an optional note after a colon.

An H2 named Optional marks links an agent can skip when short on context. The links are meant to point at Markdown versions of pages, usually at the same URL with .md added. Version 2 also lets a file sit at any path, covering the pages under that path, and suggests rel="describedby" links so clients can find it.

The proposal is explicit about the job the file does: “robots.txt lets automated tools know what access to a site is considered acceptable”, while “llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user.” It grants no permissions and blocks nothing. To control access, you still need robots.txt.

Does any AI company say it reads your llms.txt?

No, not on any page we could find. We read each operator’s crawler documentation on 8 November 2026 and searched it for the file name. Three of the four say nothing about reading third-party files, and the fourth says it does not.

Google is the only one with a stated position, and the position covers Google Search. Its guide adds: “It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” That page was last updated on 10 July 2026.

Two things are easy to misread here. A company publishing a file for its own developer documentation shows that the format works for docs. It says nothing about whether that company’s crawlers look for yours. The proposal draws the same line when it notes that the AI labs “publish llms.txt files for their own developer docs”.

And Chrome’s Lighthouse has an llms.txt audit, in its agentic browsing checks, describing the file as “an emerging convention”. The audit flags a server error, and treats a 404 as not applicable, “as providing the file is optional at the moment”. That is a tool checking a URL, and it is a different team from Search.

Silence is not a denial. We read the pages linked above, not every document each company has published, and none of this rules out a product quietly fetching the file. It means that if someone sells you llms.txt as a way to get cited, there is no operator statement behind the claim.

Do AI bots actually fetch llms.txt?

Some do, in small numbers. On our own site, invisit.dev, Claude-User requested /llms.txt 3 times in the 30 days to 8 November 2026, out of 76 Claude-User requests in total, and /llms-full.txt once. PerplexityBot, GPTBot and ClaudeBot each requested /llms.txt once. GPTBot and ClaudeBot were walking the whole site at the time, so one request each tells you about a crawler visiting URLs, and little about the file. ChatGPT-User and OAI-SearchBot did not request it at all.

Treat this as one anecdote with numbers attached:

• It is a single site, and logging began on 4 November, so the real window is days.
• Some fetches may be our own team’s agents.
• Our site points at the file from robots.txt and from the Markdown copies of its pages, so we advertise it harder than most sites would.
• A fetch shows the file was requested. It does not show that an answer used it.

To count the same thing on your own site, read your server logs. This works on the nginx or Apache combined format and also matches llms-full.txt:

awk -F'"' '$2 ~ /^(GET|HEAD) \/llms(-full)?\.txt/ {print $6}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn

It prints one line per user agent with its request count. Browser analytics cannot see any of this, because an agent fetching a text file runs no script.

Should you add an llms.txt file?

Add one if agents are likely to use your documentation and writing it costs you little. Skip it if you are hoping for rankings or citations, because nothing supports that.

That last row is more common than you would expect. Of the 2,002 sites with an llms.txt file and a readable robots.txt, 133 block GPTBot by name and 25 block all of ChatGPT-User, Claude-User and Perplexity-User. In 147 of them, robots.txt disallows /llms.txt itself for at least one of the eight crawlers and fetchers we tested, usually because a blanket block covers the path too.

The real cost is upkeep. A file that lists retired pages or old prices gives agents a confident wrong answer, which is worse than giving them nothing and letting them read your site. If nobody will maintain it, generate it from your CMS or leave it out.

What should an llms.txt file look like?

A good file is short, written by someone who knows what matters, and links to pages that read cleanly as text. This one follows the v2 format. We parsed it with llms_txt, the Python package by the proposal’s author, which pulled out the title, summary, detail text and all six links:

# Acme Payments

> Acme Payments is a card processing API for European online shops. These links cover setup, the API and billing.

Prices are in euros. The API is versioned by date, and the current version is 2026-10-01.

## Docs

- [Quickstart](https://example.com/docs/quickstart.md): Create a key and take a first test payment
- [API reference](https://example.com/docs/api.md): Every endpoint, parameter and error code
- [Webhooks](https://example.com/docs/webhooks.md): Event types, signatures and retries

## Pricing and policies

- [Pricing](https://example.com/pricing.md): Per-transaction fees by card type
- [Refund policy](https://example.com/legal/refunds.md): Time limits and who pays the fee

## Optional

- [Changelog](https://example.com/changelog.md): API changes by date

Serve it at the root as plain text, keep the opening # on the first line, and link to real .md URLs only if they exist. Then check what you actually serve, since a default 200 from your framework is the most common way to get this wrong:

curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.com/llms.txt
# 200 text/plain; charset=utf-8
curl -s https://example.com/llms.txt | head -1
# # Acme Payments

If the content type is text/html, or the first line is <!doctype html>, you are serving a page and not the file. Also make sure /llms.txt is allowed in robots.txt for the agents you want reading it.

How can you see whether agents read yours?

Your access logs are the free answer, using the command above. invisit, the tool we build, records the same requests if you run its server SDK: @invisitdev/server keeps crawler requests for any path that is not a static asset, so /llms.txt shows up, and clicking a bot on the Crawlers page lists the paths it read (crawler reads). It tells you a bot asked for the file and nothing about what the assistant did with it, and it identifies bots by user agent without checking IP lists, so a fake Claude-User counts as Claude-User. Its free AI visibility check fetches /llms.txt and checks for a title and links, and reports the file as optional for the same reason this post does.

For which bots to expect and how to tell real ones from fakes, see AI crawler user agents.

To see whether people and agents actually use the docs behind that file, see documentation analytics.