What a 10M-page site shows about links and crawl budget

Crawl budget usually gets discussed in the abstract: Googlebot has finite time and finite requests, so spend them wisely. Botify’s published case study of a large classified-style site puts real numbers on what happens when the site’s internal linking is the thing actually blocking that budget from reaching most of the pages that exist.

What was actually broken#

The site, a US online vehicle marketplace, had roughly 10 million pages at the start of the engagement. Botify’s own figures put 9.9 million of them, 99% of the total, as never having been crawled by Google at all. The cause Botify identifies is not an algorithmic penalty or a technical block, it is structural: more than 1.3 million indexable pages had exactly one internal link pointing to them, and only 2% of the URLs Google had crawled were actually inside the site’s real navigation structure. Most of what little crawl budget the site received was going to URLs outside the pages that mattered, while the vehicle listing pages that should have driven organic traffic sat effectively unreachable by a normal crawl path.

Expired vehicle listings compounded the problem from a different angle, generating redirects that lowered the share of indexable pages and adding more low-value URLs to what Google encountered when it did crawl the site.

What changed, and what moved#

Botify’s team made three categories of change: robots.txt updates to block non-critical sort-and-filter URL variations, internal linking changes that added homepage links to vehicle categories and restructured breadcrumbs to follow the Make, Model, City path a real customer takes, and a cleaned sitemap that dropped expired listings.

Measured over a six-week pilot window, known URLs fell by roughly half, the average number of inlinks per page rose 15%, and crawl activity to the site’s strategic pages rose 19 times over where it started. Three months out, Botify reports organic search traffic had doubled.

Why we care#

The size of the swing here, a 19x increase in crawl activity, is a function of how badly broken the starting point was: a 10 million-page site where 99% of pages had never been crawled has far more room to improve than a typical site already receiving adequate crawl coverage. What transfers beyond this one engagement is the mechanism Botify names, not the multiplier: pages with only one internal link, and crawl budget going to URLs outside a site’s real structure, are both concrete, auditable conditions, not vague crawl-budget folklore. A site with either pattern present at scale has a specific, checkable reason its important pages might be under-crawled, independent of whatever multiplier its own fix would produce.