llms.txt fetch counts: what AI crawlers requested on 83 sites

Most llms.txt coverage counts how many sites publish the file. This report looks at the other side: whether the crawlers behind AI products request it at all. Our adoption tracker comparison covers the publisher side, and the W3C standardization draft covers the spec. The data here comes from one primary source, EZY Research, which read its own server logs.

How many times did AI crawlers fetch llms.txt?#

EZY Research published fetch counts for five crawlers over 12 weeks, alongside their robots.txt fetches on the same sites. The pattern is lopsided for almost every crawler.

Crawler	robots.txt fetches	llms.txt fetches
OpenAI (GPTBot)	3,990	7
Anthropic (ClaudeBot)	3,120	9
PerplexityBot	775	0
Googlebot	5,125	67
Meta-ExternalAgent	172	193

In EZY’s words, the same crawlers that pulled robots.txt thousands of times pulled llms.txt in single digits. Beyond the named crawlers, the page says generic and unverified bots fetched llms.txt 1,001 times and human visitors 564 times. The page does not state how many of the 83 sites received any fetch at all, so this report does not either.

Which crawler is the exception?#

Meta-ExternalAgent is the only crawler in the table that requested llms.txt more often than robots.txt: 193 llms.txt fetches against 172 robots.txt fetches. The source does not explain why, and neither will we. Googlebot’s 67 fetches are the next highest, but they sit against 5,125 robots.txt requests.

EZY’s own conclusion is that robots.txt is where AI access is actually decided today, and that llms.txt remains cheap, standards-track insurance. That is the publisher’s reading of its data, not a measurement of any crawler’s intent.

What are the limits of this data?#

The panel is not a random sample of the web. EZY states it skews toward small-business sites whose owners sought an AI-visibility tool, and its 83 sites are a subset of a 556-domain panel. Bots were identified by user agent with signature verification only where available, so some attribution rests on the user agent string alone. Aggregate CSVs are available from EZY on request; no public dataset link is published on the page.

Read the counts as a floor on what one panel of small sites saw over one 12-week window, not as a rate across the AI search ecosystem. A fetch count also says nothing about whether a crawler used the file’s contents in an answer.

Why we care#

If you publish llms.txt expecting named AI crawlers to read it, this panel found single-digit fetches from OpenAI and Anthropic and none from Perplexity. Treat the file as low-cost insurance, not a lever, and keep robots.txt rules as the control that crawlers demonstrably request.