--- title: "llms.txt and robots.txt have two separate, unrelated standards tracks" url: https://seomadman.com/news/w3c-llms-txt-standardization-draft section: news published: 2026-06-24T00:00:00.000Z modified: 2026-06-24T00:00:00.000Z author: Adam Hafez topics: ["AI search"] --- # llms.txt and robots.txt have two separate, unrelated standards tracks ## The short answer Blog posts this year describe a June 2026 W3C draft formalizing llms.txt and giving it precedence over robots.txt. We could not find that draft on w3.org. What W3C actually has is a tracking issue opened in April 2025, still marked an early-stage investigation. A separate, real IETF working group standardizes robots.txt-adjacent AI signals and is not about llms.txt. ## Key takeaways - Jeremy Howard of Answer.AI proposed llms.txt on 3 September 2024 as a community convention, never a W3C or IETF submission. - The W3C's only documented engagement with llms.txt is GitHub issue 506, opened 27 April 2025 and still labeled an early-stage investigation. - We could not locate the June 2026 W3C working draft that multiple blog posts describe by name, and no page on w3.org matches that description. - The IETF's AI Preferences working group is real, chartered 7 January 2025, and it standardizes preference signals attached to robots.txt, a different project from llms.txt. - Nothing here changes ranking or crawling behavior today, since neither effort has shipped a document any crawler is bound to follow. A claim has been circulating since around June 2026: the W3C has published a working draft that formalizes llms.txt, adding a versioning header, a strict markdown subset and, most notably, giving llms.txt precedence over robots.txt when the two conflict. We went looking for the draft itself. We could not find it, and the two real standards efforts nearby are not the one being described. ## What llms.txt actually is Jeremy Howard, of Answer.AI and fast.ai, proposed llms.txt on 3 September 2024. The idea is a markdown file at a site's root, `/llms.txt`, listing a site's most important pages with one-line descriptions so a language model can navigate a large site within a limited context window. Howard's own post frames it as a convention, comparable to robots.txt or sitemap.xml in spirit but not affiliated with either. It was never submitted to the W3C or the IETF. Our own count of 60 major sites found 14 serving one, concentrated among [SEO vendors and developer platforms](/studies/llms-txt-adoption-2026), not publishers. ## What the W3C has actually done with it The W3C's Technical Strategy Pipeline is where the organization logs proposals it is watching before any working group forms around them. [Issue 506 in the w3c/strategy repository](https://github.com/w3c/strategy/issues/506), opened 27 April 2025 and labeled "AI", covers llms.txt. Its own text calls the proposal "in an early stage" and notes it "does not introduce any new format." As of this writing that issue is still open, with no linked working draft, no community group charter and no Technical Report. We searched w3.org's own Technical Reports index and found nothing matching the "AI Crawler Guidance Standardization" title that several blog posts attribute to a June 2026 publication, and none of those posts link to the document they describe. We are treating that specific claim as unverified and are not repeating its details as fact. ## The IETF effort is real, and it is a different project The IETF's AI Preferences working group, AIPREF, was chartered on 7 January 2025 and has an approved charter on the [IETF datatracker](https://datatracker.ietf.org/doc/charter-ietf-aipref/). Its scope is standardizing vocabulary for AI-related content preferences, such as distinguishing crawling for search retrieval from crawling for model training, and attaching that vocabulary to robots.txt, the file defined by RFC 9309, and to HTTP headers. That work is active and citable, and we covered its two current Internet-Drafts in detail [in a separate piece](/news/ietf-aipref-robots-txt-successor). It has no connection to llms.txt's page-listing format: one project is about what a crawler is permitted to do, the other is about which pages a model should read first. ## Why we care The two efforts get blended into one story because they both touch the boundary between robots.txt and AI crawlers, and a confident blog post is easier to write than a github.com issue is to read. For a technical reader, the distinction matters: AIPREF is a chartered IETF working group building toward an RFC, while llms.txt's only W3C footprint is a single tracking issue still at the investigation stage. Neither has produced a document a crawler is obligated to follow, so nothing here changes what you should ship in [robots.txt](/tools/robots-and-llms-txt-generator) today. If a vendor cites the June 2026 W3C draft specifically, ask them for the URL: we looked and could not find one. ## Frequently asked questions ### Is llms.txt now a W3C standard? No. The only W3C artifact we could verify is a tracking issue in the w3c/strategy repository, opened in April 2025 and still marked as an early-stage investigation. That is several steps before a working draft, and far short of a Recommendation. ### Does llms.txt now override robots.txt? We found no primary source for that claim. It appears in blog posts describing an unverified June 2026 W3C draft, but no W3C working draft, community group or Technical Report matching that description exists on w3.org as of this writing. ### What is the IETF AI Preferences group actually doing? It is standardizing how AI-related preferences get attached to content and to robots.txt, things like distinguishing crawl-for-training from crawl-for-retrieval. It has nothing to do with llms.txt's page-listing format. ## Sources 1. [llms.txt (W3C Technical Strategy Pipeline tracking issue)](https://github.com/w3c/strategy/issues/506) - W3C (primary) 2. [AI Preferences (aipref) working group charter](https://datatracker.ietf.org/doc/charter-ietf-aipref/) - IETF (primary) 3. [/llms.txt: a proposal to provide information to help LLMs use websites](https://www.answer.ai/posts/2024-09-03-llmstxt.html) - Answer.AI (primary) 4. [Dataset: AI crawler policies and llms.txt across 60 sites](https://seomadman.com/data/ai-crawler-policies.json) - SEO Madman