llms.txt is a vendor standard, not a publisher one

The llms.txt proposal asks a site to publish a plain-text map of its most useful pages for language models, in the spirit of robots.txt or sitemap.xml. Two years of conference slides later, the useful question is not whether it is a good idea. It is who actually put the file on their server.

What we measured#

One request per site, counted strictly. /llms.txt on each of 60 major sites, 10 September 2026, alongside the robots.txt crawl published in our AI crawler research. A file counted only if the response was 200, the content type was not HTML, and the body was not empty - a large share of sites answer any unknown path with a styled 404 under a 200 status, and counting those would have roughly doubled the result. The per-site outcome sits in the same JSON dataset.

Who serves one#

Fourteen sites of sixty. Seven sell SEO, marketing or optimisation software: Semrush, Yoast, SEOPress, Onely, Conductor, BrightEdge and HubSpot. Five are developer or commerce platforms: Cloudflare, Vercel, GitHub, Shopify and WordPress.org. One is a trade publication, Search Engine Land. One is a news publisher, Time.

Twenty-nine of 30 news publishers serve nothing at that path, including every outlet with a licensing deal or a lawsuit in progress.

Why the split makes sense#

The adopters share a product shape, not a sophistication level. Every organisation on the list has documentation as a product surface - an API reference, a plugin manual, a help centre. A file that says “here are the canonical explanations, in order” costs an afternoon and matches something they already maintain.

A newspaper has no equivalent artifact. Its corpus is thousands of dated articles with no stable hierarchy, and the commercial instinct of the last two years has been to withhold that corpus rather than index it more helpfully for machines. Our robots.txt crawl found 24 of 28 reachable news publishers blocking at least one AI crawler outright - a site trying to keep models out has no reason to hand them a table of contents.

Why we care#

If you publish documentation, llms.txt is a reasonable, cheap addition - the companies most practised at documentation adopted it first for exactly that reason.

If you publish news, it is not the highest-value item on your list. Nothing in this data suggests it changes how often you are cited; structured data, clean canonicals and a machine-readable version of the article itself do more, because those are surfaces search engines have actually committed to reading.

If a vendor claims llms.txt affects AI visibility, ask for the commitment they are relying on. No major engine has published one. The file is a convention with an adoption base of fourteen sites in sixty, and saying that plainly beats treating it as settled practice in technical SEO.