Who blocks AI crawlers: 60 robots.txt files, read in full

Every publisher in this sample has had two years to decide what to do about AI crawlers. On 10 September 2026 we asked all 60 of them the same question, in the only place the answer is written down: robots.txt.

What we did#

One request per site, parsed by the rules. A single GET of /robots.txt, grouped the way RFC 9309 defines it, then one question per agent: does a group naming this crawler disallow the whole site. Fifteen agents checked, from GPTBot and ClaudeBot to Bytespider and Applebot-Extended.

Fifty-seven of sixty files were readable. Politico (403), NPR (connection failure) and Stack Overflow (418) are excluded from the denominators, not quietly counted as open. The full per-site result is published as JSON, so every number below can be checked against the file it came from.

What the files say#

Twenty-nine of 57 sites block at least one named AI crawler site-wide. Thirty-three name at least one, so a handful name agents only to allow them, or to restrict them to part of the site.

The most-blocked agents cluster near the top. ClaudeBot and Bytespider are each blocked by 27 sites, CCBot by 25, Applebot-Extended by 24, PerplexityBot by 22. GPTBot, the name that gets the most attention, sits lower at 17 blocks from 24 sites that name it.

Google-Extended is the outlier. Twenty sites name it, 16 block it - the only training-related agent with a meaningful gap between being noticed and being refused. It shares infrastructure with Search traffic, and publishers have accepted terms from Google for two decades that they refuse from anyone else.

The finding worth arguing about#

Split by who runs the site and the picture stops being a spectrum. Of 28 reachable news publishers, 24 block at least one AI crawler, including the New York Times, the Washington Post, the Guardian, the BBC, AP, Bloomberg, the FT, CNN, Wired, Ars Technica, the Economist and Nature.

Of 15 SEO trade sites, 2 do. Search Engine Land, Search Engine Journal, Search Engine Roundtable, Moz, Ahrefs, Semrush, Yoast, SEOPress, Screaming Frog, Sistrix, Conductor and BrightEdge all leave every named AI agent free to take the whole site. The industry that publishes the most words about AI search is the industry least likely to withhold its own text from it - a coherent strategy for a vendor selling software, and a harder one to defend from an industry advising publishers on a trade it has not made itself.

The distinction almost nobody is drawing#

OpenAI separates training from retrieval; almost no one uses that. GPTBot collects training data, OAI-SearchBot handles search retrieval, ChatGPT-User fetches what a person asked for - a publisher can decline training while staying citable in answers. Nine sites in this sample block both anyway. One, Forbes, blocks GPTBot and lets OAI-SearchBot through. OAI-SearchBot is named by only 13 sites at all, against 24 for GPTBot.

Why we care#

If your robots.txt blocks both a training agent and its matching retrieval agent, confirm that is what you meant. The cost is not paid in training corpora, already assembled either way - it is paid in the retrieval layer that decides which sources an answer cites this week, a live traffic question in AI search. Read your file, not your memory of it, decide the training and retrieval questions separately, and record the decision somewhere a colleague will find it two years from now.