llms.txt and robots.txt have two separate, unrelated standards tracks

A claim has been circulating since around June 2026: the W3C has published a working draft that formalizes llms.txt, adding a versioning header, a strict markdown subset and, most notably, giving llms.txt precedence over robots.txt when the two conflict. We went looking for the draft itself. We could not find it, and the two real standards efforts nearby are not the one being described.

What llms.txt actually is#

Jeremy Howard, of Answer.AI and fast.ai, proposed llms.txt on 3 September 2024. The idea is a markdown file at a site’s root, /llms.txt, listing a site’s most important pages with one-line descriptions so a language model can navigate a large site within a limited context window. Howard’s own post frames it as a convention, comparable to robots.txt or sitemap.xml in spirit but not affiliated with either. It was never submitted to the W3C or the IETF. Our own count of 60 major sites found 14 serving one, concentrated among SEO vendors and developer platforms, not publishers.

What the W3C has actually done with it#

The W3C’s Technical Strategy Pipeline is where the organization logs proposals it is watching before any working group forms around them. Issue 506 in the w3c/strategy repository, opened 27 April 2025 and labeled “AI”, covers llms.txt. Its own text calls the proposal “in an early stage” and notes it “does not introduce any new format.” As of this writing that issue is still open, with no linked working draft, no community group charter and no Technical Report. We searched w3.org’s own Technical Reports index and found nothing matching the “AI Crawler Guidance Standardization” title that several blog posts attribute to a June 2026 publication, and none of those posts link to the document they describe. We are treating that specific claim as unverified and are not repeating its details as fact.

The IETF effort is real, and it is a different project#

The IETF’s AI Preferences working group, AIPREF, was chartered on 7 January 2025 and has an approved charter on the IETF datatracker. Its scope is standardizing vocabulary for AI-related content preferences, such as distinguishing crawling for search retrieval from crawling for model training, and attaching that vocabulary to robots.txt, the file defined by RFC 9309, and to HTTP headers. That work is active and citable, and we covered its two current Internet-Drafts in detail in a separate piece. It has no connection to llms.txt’s page-listing format: one project is about what a crawler is permitted to do, the other is about which pages a model should read first.

Why we care#

The two efforts get blended into one story because they both touch the boundary between robots.txt and AI crawlers, and a confident blog post is easier to write than a github.com issue is to read. For a technical reader, the distinction matters: AIPREF is a chartered IETF working group building toward an RFC, while llms.txt’s only W3C footprint is a single tracking issue still at the investigation stage. Neither has produced a document a crawler is obligated to follow, so nothing here changes what you should ship in robots.txt today. If a vendor cites the June 2026 W3C draft specifically, ask them for the URL: we looked and could not find one.