--- title: "llms.txt is a vendor standard, not a publisher one" url: https://seomadman.com/studies/llms-txt-adoption-2026 section: studies published: 2026-09-10T00:00:00.000Z modified: 2026-09-10T00:00:00.000Z author: Adam Hafez topics: ["AI search", "Technical SEO"] --- # llms.txt is a vendor standard, not a publisher one ## The short answer We requested /llms.txt from 60 major sites on 10 September 2026. Fourteen served a real file. Seven of those are SEO or marketing vendors and only one is a news publisher, so adoption is concentrated in the companies that sell tooling rather than the ones whose content answer engines actually quote. ## Key takeaways - Fourteen of 60 sites served a valid llms.txt, meaning a 200 response with text content rather than an HTML error page. - Seven of the 14 are SEO or marketing vendors, including Semrush, Yoast, SEOPress, Onely, Conductor, BrightEdge and HubSpot. - Only one of 30 news publishers in the sample serves one, which is Time. - Developer platforms are the other adopter group, with Cloudflare, Vercel, GitHub, Shopify and WordPress.org all serving a file. - No search engine has committed to reading llms.txt, so today the file is a publishing convention rather than a ranking or retrieval input. The llms.txt proposal asks a site to publish a plain-text map of its most useful pages for language models, in the spirit of robots.txt or sitemap.xml. Two years of conference slides later, the useful question is not whether it is a good idea. It is who actually put the file on their server. ## What we measured **One request per site, counted strictly.** `/llms.txt` on each of 60 major sites, 10 September 2026, alongside the robots.txt crawl published in our [AI crawler research](/research/ai-crawler-policies-news-sites). A file counted only if the response was 200, the content type was not HTML, and the body was not empty - a large share of sites answer any unknown path with a styled 404 under a 200 status, and counting those would have roughly doubled the result. The per-site outcome sits in the same [JSON dataset](https://seomadman.com/data/ai-crawler-policies.json). ## Who serves one **Fourteen sites of sixty.** Seven sell SEO, marketing or optimisation software: Semrush, Yoast, SEOPress, Onely, Conductor, BrightEdge and HubSpot. Five are developer or commerce platforms: Cloudflare, Vercel, GitHub, Shopify and WordPress.org. One is a trade publication, Search Engine Land. One is a news publisher, Time. **Twenty-nine of 30 news publishers serve nothing at that path**, including every outlet with a licensing deal or a lawsuit in progress. ## Why the split makes sense **The adopters share a product shape, not a sophistication level.** Every organisation on the list has documentation as a product surface - an API reference, a plugin manual, a help centre. A file that says "here are the canonical explanations, in order" costs an afternoon and matches something they already maintain. **A newspaper has no equivalent artifact.** Its corpus is thousands of dated articles with no stable hierarchy, and the commercial instinct of the last two years has been to withhold that corpus rather than index it more helpfully for machines. Our robots.txt crawl found 24 of 28 reachable news publishers blocking at least one AI crawler outright - a site trying to keep models out has no reason to hand them a table of contents. ## Why we care **If you publish documentation, llms.txt is a reasonable, cheap addition** - the companies most practised at documentation adopted it first for exactly that reason. **If you publish news, it is not the highest-value item on your list.** Nothing in this data suggests it changes how often you are cited; structured data, clean canonicals and a machine-readable version of the article itself do more, because those are surfaces search engines have actually committed to reading. **If a vendor claims llms.txt affects AI visibility, ask for the commitment they are relying on.** No major engine has published one. The file is a convention with an adoption base of fourteen sites in sixty, and saying that plainly beats treating it as settled practice in [technical SEO](/topics/technical-seo). ## Frequently asked questions ### Does llms.txt help me rank in AI answers? There is no published commitment from Google, Microsoft, OpenAI or Anthropic to read llms.txt, so nothing in this study supports a ranking or citation claim. Treat it as documentation you control, not as a retrieval input. ### How did you decide a file counted? A request to /llms.txt had to return 200 with a non-HTML content type and a non-empty body. That excludes the common case of a site returning its styled 404 page with a 200 status, which would otherwise inflate the count. ### Should I publish one anyway? If you maintain developer or product documentation, it is cheap and it is a reasonable place to point a machine reader at your canonical explanations. If you are a publisher weighing it against basic structured data and clean markup, it is not the thing to do first. ## Sources 1. [Dataset: AI crawler policies and llms.txt across 60 sites](https://seomadman.com/data/ai-crawler-policies.json) - SEO Madman (primary) 2. [The llms.txt proposal](https://llmstxt.org/) - llmstxt.org (primary)