---
title: "AI training now drives 52% of crawler requests, Cloudflare says"
url: https://seomadman.com/reports/cloudflare-ai-training-crawler-share-2026
section: reports
published: 2026-09-18T00:00:00.000Z
modified: 2026-09-18T00:00:00.000Z
author: Adam Hafez
topics: ["AI search", "Technical SEO"]
---

# AI training now drives 52% of crawler requests, Cloudflare says

## The short answer

Cloudflare reports that 52% of crawler requests it identifies by purpose were for AI training as of June 2026, up from 22% in Spring 2025. Mixed-use crawlers add over 36% more. Pure search crawling, the kind that can send readers back to a site, is now a small and declining share of crawler activity.

## Key takeaways

- Cloudflare says 52% of crawler requests were for AI training as of June 2026, up from 22% in Spring 2025.
- Mixed-use crawlers, which blend search, agent use and training, account for over 36% of crawler activity.
- Cloudflare describes pure search crawling as a small and declining share, despite remaining critical for publisher visibility.
- More than one-third of crawler activity still comes from mixed-use bots, so purpose labels cannot fully separate training from search.
- Cloudflare defines its crawl-to-refer ratio as HTML requests from a platform's crawlers divided by HTML requests referred from that platform, normalized to one referral.

Most crawling headlines count bots. Cloudflare's July 2026 report does something more useful for
an SEO: it splits crawler requests by what the crawler is for. On that split, [AI search](/topics/ai-search)
visibility is the smaller half of the story, because most requests are not asking to send anyone
back.

## What share of crawler requests are for AI training?

Cloudflare writes that "52% of crawler requests are now for AI training as of June 2026, up from
22% in Spring 2025." It also says mixed-use crawlers, those "blending search, agent use, and
training", represent over 36% of activity, and that pure search crawling is now "a small and
declining share of overall crawler activity, despite remaining critical for publisher visibility."

Read together, the two named categories cover at least 88% of identified crawler requests. Cloudflare
does not publish the remaining split on the page, so this report does not estimate it.

## Why do mixed-use crawlers muddy the number?

Cloudflare says more than one-third of crawler activity on its network still comes from mixed-use
bots. A crawler that fetches a page for both training and answer retrieval cannot be counted as
either, so the 52% training figure is a floor for training-related fetching, not a clean ceiling.
For [technical SEO](/topics/technical-seo) work, that means a single robots.txt rule for such a
bot blocks or allows training and retrieval together.

## How does Cloudflare measure crawl-to-refer ratio?

The ratio comes from a separate Radar feature. Cloudflare divides "the total number of requests from
relevant user agents associated with a given search or AI platform where the response was of
Content-type: text/html" by the number of HTML requests whose Referer header names that platform,
normalized to a single referral. Cloudflare notes that referrals from Google's native apps carry no
Referer header, so ratios for some platforms can read higher than reality.

## What this report does not show

The 2026 post does not give a crawl-to-refer ratio, an absolute request count or a sample size for
the purpose split, so none appear here. Cloudflare also reports that some of the most heavily
crawled categories have seen human traffic decline as much as 40% in less than one year, but the
page does not tie that figure to the purpose split, and this report does not either.

## Sources

1. [Content Independence Day, one year on: building the business model for the agentic Internet](https://blog.cloudflare.com/agentic-internet-bot-report/) - Cloudflare Blog (primary)
2. [The crawl before the fall... of referrals: understanding AI's impact on content providers](https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/) - Cloudflare Blog (primary)