--- title: "Who blocks AI crawlers: 60 robots.txt files, read in full" url: https://seomadman.com/research/ai-crawler-policies-news-sites section: research published: 2026-09-10T00:00:00.000Z modified: 2026-09-10T00:00:00.000Z author: Adam Hafez topics: ["AI search", "Technical SEO", "Indexing"] --- # Who blocks AI crawlers: 60 robots.txt files, read in full ## The short answer We fetched robots.txt from 60 publishers and SEO vendors on 10 September 2026. Of 28 reachable news publishers, 24 block at least one named AI crawler. Of 15 SEO trade sites, 2 do. The industry that publishes most about AI search is the one leaving it most open. ## Key takeaways - Of 57 sites with a reachable robots.txt, 33 name at least one AI crawler and 29 block at least one outright. - News publishers block almost as a rule: 24 of 28 reachable ones disallow a named AI agent across the whole site. - SEO trade publications behave in the opposite way, with only 2 of 15 blocking any AI crawler by name. - ClaudeBot and Bytespider are the most blocked agents at 27 sites each, followed by CCBot at 25 and Applebot-Extended at 24. - Almost nobody separates training from retrieval: 9 sites block both GPTBot and OAI-SearchBot, and only Forbes blocks the training agent while leaving the search agent through. Every publisher in this sample has had two years to decide what to do about AI crawlers. On 10 September 2026 we asked all 60 of them the same question, in the only place the answer is written down: robots.txt. ## What we did **One request per site, parsed by the rules.** A single GET of `/robots.txt`, grouped the way [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) defines it, then one question per agent: does a group naming this crawler disallow the whole site. Fifteen agents checked, from GPTBot and ClaudeBot to Bytespider and Applebot-Extended. **Fifty-seven of sixty files were readable.** Politico (403), NPR (connection failure) and Stack Overflow (418) are excluded from the denominators, not quietly counted as open. The full per-site result is published as [JSON](https://seomadman.com/data/ai-crawler-policies.json), so every number below can be checked against the file it came from. ## What the files say **Twenty-nine of 57 sites block at least one named AI crawler site-wide.** Thirty-three name at least one, so a handful name agents only to allow them, or to restrict them to part of the site. **The most-blocked agents cluster near the top.** ClaudeBot and Bytespider are each blocked by 27 sites, CCBot by 25, Applebot-Extended by 24, PerplexityBot by 22. GPTBot, the name that gets the most attention, sits lower at 17 blocks from 24 sites that name it. **Google-Extended is the outlier.** Twenty sites name it, 16 block it - the only training-related agent with a meaningful gap between being noticed and being refused. It shares infrastructure with Search traffic, and publishers have accepted terms from Google for two decades that they refuse from anyone else. ## The finding worth arguing about **Split by who runs the site and the picture stops being a spectrum.** Of 28 reachable news publishers, 24 block at least one AI crawler, including the New York Times, the Washington Post, the Guardian, the BBC, AP, Bloomberg, the FT, CNN, Wired, Ars Technica, the Economist and Nature. **Of 15 SEO trade sites, 2 do.** Search Engine Land, Search Engine Journal, Search Engine Roundtable, Moz, Ahrefs, Semrush, Yoast, SEOPress, Screaming Frog, Sistrix, Conductor and BrightEdge all leave every named AI agent free to take the whole site. The industry that publishes the most words about AI search is the industry least likely to withhold its own text from it - a coherent strategy for a vendor selling software, and a harder one to defend from an industry advising publishers on a trade it has not made itself. ## The distinction almost nobody is drawing **OpenAI separates training from retrieval; almost no one uses that.** GPTBot collects training data, OAI-SearchBot handles search retrieval, ChatGPT-User fetches what a person asked for - a publisher can decline training while staying citable in answers. Nine sites in this sample block both anyway. One, Forbes, blocks GPTBot and lets OAI-SearchBot through. OAI-SearchBot is named by only 13 sites at all, against 24 for GPTBot. ## Why we care If your robots.txt blocks both a training agent and its matching retrieval agent, confirm that is what you meant. The cost is not paid in training corpora, already assembled either way - it is paid in the retrieval layer that decides which sources an answer cites this week, a live traffic question in [AI search](/topics/ai-search). Read your file, not your memory of it, decide the training and retrieval questions separately, and record the decision somewhere a colleague will find it two years from now. ## Frequently asked questions ### Does blocking GPTBot remove my site from ChatGPT? Not by itself. GPTBot is the crawler associated with training data collection, while OAI-SearchBot and ChatGPT-User relate to search retrieval and user-initiated fetches. In this sample 9 sites block both and only one blocks the training agent alone, which means most sites making the block are making a broader decision than they may realise. ### Why measure only site-wide blocks? Because a partial rule is a judgement call and a site-wide disallow is not. Counting "Disallow: /" under a named agent produces a number anyone can reproduce from the same file, which is the point of publishing the dataset next to the finding. ### Can I check your numbers? Yes. The full per-site dataset, including the raw decision for every agent on every host, is published as JSON at the link in this article, with the collection timestamp inside it. ## Sources 1. [Dataset: AI crawler policies across 60 sites (this study)](https://seomadman.com/data/ai-crawler-policies.json) - SEO Madman (primary) 2. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) - IETF (primary) 3. [Google-Extended and Search crawler documentation](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) - Google Search Central (primary)