AI crawler block rate: how often a 403 actually stops the bot

“AI crawlers are getting blocked more” is easy to say and hard to measure, because two entirely different things get called “blocking”: a line in robots.txt asking a crawler to stay away, and a server actually returning HTTP 403 when the crawler shows up anyway. Our own read of 60 robots.txt files found who writes the rule. This report tracks the other half - how often the request itself gets refused, and how that rate has moved over the past year.

The rate that measures the actual request#

TechnologyChecker.io draws on Cloudflare Radar’s bot response-status data, which classifies traffic from crawlers Cloudflare has verified against each operator’s published IP ranges and records the live HTTP status returned to each request. In the first half of 2026, 7.9% of those verified AI crawler requests came back HTTP 403 Forbidden, with a further 0.7% rate-limited at 429

• roughly one in twelve AI crawl attempts actively refused, before counting the ones that were merely slowed down.

Widen the lens to all 4xx responses - forbidden, rate-limited, or hitting a dead URL - and the trend is sharper still. Clean 200 OK responses to crawler requests fell from 80.5% in Q2 2025 to 49.2% in Q2 2026, while 4xx responses climbed from 10.2% to 35.8% over the same stretch. More than one in three AI crawler requests now hits some kind of block, a rate that has more than tripled in a year on the same network, measured the same way each time.

Disallow rules count something different#

A second TechnologyChecker.io dataset - a text scan of 4,047 robots.txt files as of Q1 2026, growing to 4,223 by July - answers a different question: which crawlers get named in a disallow rule at all, not whether any of their requests were ever refused. In that Q1 2026 sample, GPTBot appeared in 5.52% of disallow rules, ahead of CCBot (5.08%), ClaudeBot (4.88%), Google-Extended (4.44%) and Bytespider (4.23%).

That figure cannot be read against the 403 rate above as if they measured the same thing. A disallow rule is a published intention a crawler can simply ignore - our robots.txt read of 60 publishers found exactly that gap between intention and what a site actually enforces at the edge. A 403 is enforced regardless of whether the crawler ever fetched the rules file. The two trackers publish their methodology separately for this reason, and treating a disallow-rule share as a blocked-request rate - or vice versa - overstates or understates what is actually happening at the server.

Cloudflare changed the default, then published the count#

The infrastructure context for both numbers: on 1 July 2025, Cloudflare’s own “Content Independence Day” post changed the default so AI crawlers are blocked unless a site owner opts them in, and the company’s one-year-later report confirms this applies to new domains by default. Existing domains kept whatever setting they already had. Cloudflare CEO Matthew Prince said at WIRED’s Big Interview event in December 2025 that customers had blocked more than 416 billion AI bot scraping requests in the five months since that policy took effect - a count reported by Search Engine Land, Computerworld and others, though not itself the subject of a dedicated Cloudflare blog post at that point.

Why we care#

A rising 403 rate and a rising disallow-rule count both describe more resistance to AI crawlers, but they are not the same fact and should not be quoted as if they were. If a claim cites a percentage of AI bot requests being blocked, ask whether the source measured live response codes or counted robots.txt rules - the gap between the two, visible in our own robots.txt research, is often the difference between a policy that exists on paper and one a server actually enforces.

The trend, on the metric that measures enforcement, is unambiguous: it is rising, fast. A tripling of 4xx responses to AI crawlers within a year, on the same network, measured the same way each period, is the closest thing in this space to a real trend line rather than a single snapshot

• the caveat is that it describes Cloudflare’s network, not the open web as a whole, the same population caveat that applies to any single-vendor dataset in AI search measurement.