---
title: "llms.txt fetch counts: what AI crawlers requested on 83 sites"
url: https://seomadman.com/reports/llms-txt-fetch-counts-by-ai-crawler
section: reports
published: 2026-09-18T00:00:00.000Z
modified: 2026-09-18T00:00:00.000Z
author: Adam Hafez
topics: ["AI search"]
---

# llms.txt fetch counts: what AI crawlers requested on 83 sites

## The short answer

Server logs from 83 websites over 12 weeks show major AI crawlers barely touch llms.txt: OpenAI's crawler fetched it 7 times against 3,990 robots.txt fetches, Anthropic's 9 against 3,120, and PerplexityBot never. Meta-ExternalAgent is the exception at 193 llms.txt fetches, more than its 172 robots.txt fetches.

## Key takeaways

- OpenAI's crawler fetched llms.txt 7 times across 83 sites in 12 weeks, against 3,990 robots.txt fetches from the same crawler.
- Anthropic's ClaudeBot fetched llms.txt 9 times versus 3,120 robots.txt fetches, and PerplexityBot fetched it zero times versus 775.
- Googlebot requested llms.txt 67 times, roughly a hundredth of the 5,125 times it requested robots.txt.
- Meta-ExternalAgent is the one crawler that fetched llms.txt more than robots.txt, with 193 llms.txt requests against 172.
- The panel skews toward small-business sites whose owners sought an AI-visibility tool, so the counts describe that population and not the whole web.

Most llms.txt coverage counts how many sites publish the file. This report looks at the other side: whether the crawlers behind AI products request it at all. Our [adoption tracker comparison](/reports/llms-txt-adoption-tracker-comparison) covers the publisher side, and the [W3C standardization draft](/news/w3c-llms-txt-standardization-draft) covers the spec. The data here comes from one primary source, EZY Research, which read its own server logs.

## How many times did AI crawlers fetch llms.txt?

EZY Research published fetch counts for five crawlers over 12 weeks, alongside their robots.txt fetches on the same sites. The pattern is lopsided for almost every crawler.

| Crawler | robots.txt fetches | llms.txt fetches |
|---|---|---|
| OpenAI (GPTBot) | 3,990 | 7 |
| Anthropic (ClaudeBot) | 3,120 | 9 |
| PerplexityBot | 775 | 0 |
| Googlebot | 5,125 | 67 |
| Meta-ExternalAgent | 172 | 193 |

In EZY's words, the same crawlers that pulled robots.txt thousands of times pulled llms.txt in single digits. Beyond the named crawlers, the page says generic and unverified bots fetched llms.txt 1,001 times and human visitors 564 times. The page does not state how many of the 83 sites received any fetch at all, so this report does not either.

## Which crawler is the exception?

Meta-ExternalAgent is the only crawler in the table that requested llms.txt more often than robots.txt: 193 llms.txt fetches against 172 robots.txt fetches. The source does not explain why, and neither will we. Googlebot's 67 fetches are the next highest, but they sit against 5,125 robots.txt requests.

EZY's own conclusion is that robots.txt is where AI access is actually decided today, and that llms.txt remains cheap, standards-track insurance. That is the publisher's reading of its data, not a measurement of any crawler's intent.

## What are the limits of this data?

The panel is not a random sample of the web. EZY states it skews toward small-business sites whose owners sought an AI-visibility tool, and its 83 sites are a subset of a 556-domain panel. Bots were identified by user agent with signature verification only where available, so some attribution rests on the user agent string alone. Aggregate CSVs are available from EZY on request; no public dataset link is published on the page.

Read the counts as a floor on what one panel of small sites saw over one 12-week window, not as a rate across the [AI search](/topics/ai-search) ecosystem. A fetch count also says nothing about whether a crawler used the file's contents in an answer.

## Why we care

**If you publish llms.txt expecting named AI crawlers to read it, this panel found single-digit fetches from OpenAI and Anthropic and none from Perplexity.** Treat the file as low-cost insurance, not a lever, and keep robots.txt rules as the control that crawlers demonstrably request.

## Sources

1. [We put llms.txt on 83 websites. In 12 weeks, OpenAI's crawler read it 7 times.](https://www.ezy.ai/research/do-ai-bots-read-llms-txt) - EZY Research (primary)