---
title: "AI crawler block rate: how often a 403 actually stops the bot"
url: https://seomadman.com/reports/ai-crawler-blocking-rate-trend-2026
section: reports
published: 2026-06-17T00:00:00.000Z
modified: 2026-06-17T00:00:00.000Z
author: Adam Hafez
topics: ["AI search"]
---

# AI crawler block rate: how often a 403 actually stops the bot

## The short answer

Across Cloudflare's network, AI crawler requests returning HTTP 403 Forbidden ran at 7.9% in the first half of 2026, with overall 4xx responses to crawlers climbing from 10.2% in Q2 2025 to 35.8% by Q2 2026. That is a live response-code measurement, distinct from counting how many robots.txt files merely name a crawler in a disallow rule.

## Key takeaways

- In H1 2026, 7.9% of verified AI crawler requests on Cloudflare's network returned HTTP 403 Forbidden, plus 0.7% rate-limited with 429.
- Overall 4xx responses to crawler requests rose from 10.2% in Q2 2025 to 35.8% in Q2 2026, while clean 200 OK responses fell from 80.5% to 49.2% over the same span.
- A disallow rule and a 403 measure different things: a Q1 2026 scan of 4,047 robots.txt files found GPTBot named in 5.52% of disallow rules, ahead of CCBot and ClaudeBot.
- Cloudflare made blocking AI crawlers the default for new domains from 1 July 2025, and says customers blocked over 416 billion AI bot requests in the policy's first five months.
- A robots.txt disallow is a request a crawler can ignore, while a 403 is enforced at the edge regardless of whether the bot ever reads the rules file.

"AI crawlers are getting blocked more" is easy to say and hard to measure, because two entirely
different things get called "blocking": a line in robots.txt asking a crawler to stay away, and a
server actually returning HTTP 403 when the crawler shows up anyway. Our own read of 60 robots.txt
files [found who writes the rule](/research/ai-crawler-policies-news-sites). This report tracks
the other half - how often the request itself gets refused, and how that rate has moved over the
past year.

## The rate that measures the actual request

TechnologyChecker.io draws on Cloudflare Radar's bot response-status data, which classifies
traffic from crawlers Cloudflare has verified against each operator's published IP ranges and
records the live HTTP status returned to each request. In the first half of 2026, 7.9% of those
verified AI crawler requests came back HTTP 403 Forbidden, with a further 0.7% rate-limited at 429
- roughly one in twelve AI crawl attempts actively refused, before counting the ones that were
merely slowed down.

Widen the lens to all 4xx responses - forbidden, rate-limited, or hitting a dead URL - and the
trend is sharper still. Clean 200 OK responses to crawler requests fell from 80.5% in Q2 2025 to
49.2% in Q2 2026, while 4xx responses climbed from 10.2% to 35.8% over the same stretch. More than
one in three AI crawler requests now hits some kind of block, a rate that has more than tripled in
a year on the same network, measured the same way each time.

## Disallow rules count something different

A second TechnologyChecker.io dataset - a text scan of 4,047 robots.txt files as of Q1 2026,
growing to 4,223 by July - answers a different question: which crawlers get named in a disallow
rule at all, not whether any of their requests were ever refused. In that Q1 2026 sample, GPTBot
appeared in 5.52% of disallow rules, ahead of CCBot (5.08%), ClaudeBot (4.88%), Google-Extended
(4.44%) and Bytespider (4.23%).

That figure cannot be read against the 403 rate above as if they measured the same thing. A
disallow rule is a published intention a crawler can simply ignore - our robots.txt read of 60
publishers found [exactly that gap between intention and](/research/ai-crawler-policies-news-sites)
what a site actually enforces at the edge. A 403 is enforced regardless of whether the crawler
ever fetched the rules file. The two trackers publish their methodology separately for this
reason, and treating a disallow-rule share as a blocked-request rate - or vice versa - overstates
or understates what is actually happening at the server.

## Cloudflare changed the default, then published the count

The infrastructure context for both numbers: on 1 July 2025, Cloudflare's own "Content Independence
Day" post changed the default so AI crawlers are blocked unless a site owner opts them in, and the
company's one-year-later report confirms this applies to new domains by default. Existing domains
kept whatever setting they already had. Cloudflare CEO Matthew Prince said at WIRED's Big Interview
event in December 2025 that customers had blocked more than 416 billion AI bot scraping requests
in the five months since that policy took effect - a count reported by Search Engine Land,
Computerworld and others, though not itself the subject of a dedicated Cloudflare blog post at that
point.

## Why we care

**A rising 403 rate and a rising disallow-rule count both describe more resistance to AI crawlers,
but they are not the same fact and should not be quoted as if they were.** If a claim cites a
percentage of AI bot requests being blocked, ask whether the source measured live response codes
or counted robots.txt rules - the gap between the two, visible in our own [robots.txt
research](/research/ai-crawler-policies-news-sites), is often the difference between a policy that
exists on paper and one a server actually enforces.

**The trend, on the metric that measures enforcement, is unambiguous: it is rising, fast.** A
tripling of 4xx responses to AI crawlers within a year, on the same network, measured the same way
each period, is the closest thing in this space to a real trend line rather than a single snapshot
- the caveat is that it describes Cloudflare's network, not the open web as a whole, the same
population caveat that applies to any single-vendor dataset in [AI search](/topics/ai-search)
measurement.

## Frequently asked questions

### Does a robots.txt disallow rule mean a crawler is actually blocked?

Not by itself. A disallow rule is a published request that a well-behaved crawler chooses to honor. Whether a request was actually refused is a separate, measurable fact - the HTTP status code the server returned - and TechnologyChecker.io's two datasets track these separately rather than treating a rule as proof of a block.

### What share of AI crawler requests actually get blocked?

On Cloudflare's network, 7.9% of verified AI crawler requests returned HTTP 403 in H1 2026, with a further 0.7% rate-limited. Zoomed out further, all 4xx responses to crawlers - forbidden, rate-limited or not-found - rose from 10.2% of requests in Q2 2025 to 35.8% by Q2 2026.

### Did Cloudflare block AI crawlers by default before July 2025?

No. Cloudflare's own 1 July 2025 announcement, "Content Independence Day," changed the default so that AI crawlers are blocked unless a site owner opts them in, and its one-year-later report confirms this applies to new domains by default. Existing domains kept their prior settings unless the owner changed them.

## Sources

1. [What AI Crawlers Actually Want? (August 2026 Update)](https://technologychecker.io/blog/ai-crawler-statistics) - TechnologyChecker.io (primary)
2. [Bot Traffic Statistics 2026: How Much of the Web Is Automated](https://technologychecker.io/blog/bot-traffic-statistics) - TechnologyChecker.io
3. [We Analyzed robots.txt Across Cloudflare's Network: Publishers Now Block Training Bots and Allow Answering Bots](https://technologychecker.io/blog/robots-txt-ai-crawlers-blocking-report) - TechnologyChecker.io
4. [Content Independence Day: no AI crawl without compensation!](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/) - Cloudflare Blog (primary)
5. [Content Independence Day, one year on: building the business model for the agentic Internet](https://blog.cloudflare.com/agentic-internet-bot-report/) - Cloudflare Blog (primary)
6. [Cloudflare: 416 billion AI bot requests blocked since July](https://searchengineland.com/cloudflare-416-billion-ai-bot-requests-blocked-since-july-465704) - Search Engine Land