---
title: "What a 10M-page site shows about links and crawl budget"
url: https://seomadman.com/research/internal-linking-crawl-budget-botify-case-study
section: research
published: 2026-09-16T00:00:00.000Z
modified: 2026-09-16T00:00:00.000Z
author: Adam Hafez
topics: ["Technical SEO"]
---

# What a 10M-page site shows about links and crawl budget

## The short answer

Botify's published case study of a 10 million-page US auto marketplace found 99% of pages uncrawled by Google, largely because 1.3 million-plus pages carried only a single internal link. After restructuring internal links, breadcrumbs and the sitemap, crawl activity to strategic pages rose 19 times over six weeks, and organic traffic doubled within three months.

## Key takeaways

- Botify's case study covers a US online vehicle marketplace of roughly 10 million pages, of which 9.9 million had never been crawled by Google.
- More than 1.3 million indexable pages carried only a single internal link, a structural cause Botify identifies directly for the site's crawl gap.
- After Botify's team restructured homepage links, breadcrumbs and the sitemap, crawl activity to the site's strategic pages rose 19 times within six weeks.
- Known URLs fell by roughly half in the same six-week window, and the average number of inlinks per page rose, both measured directly rather than estimated.
- Organic search traffic doubled within three months of the internal-linking changes, per Botify's own reporting of the engagement.

Crawl budget usually gets discussed in the abstract: Googlebot has finite time and finite
requests, so spend them wisely. Botify's published case study of a large classified-style site
puts real numbers on what happens when the site's internal linking is the thing actually blocking
that budget from reaching most of the pages that exist.

## What was actually broken

The site, a US online vehicle marketplace, had roughly 10 million pages at the start of the
engagement. Botify's own figures put 9.9 million of them, 99% of the total, as never having been
crawled by Google at all. The cause Botify identifies is not an algorithmic penalty or a technical
block, it is structural: more than 1.3 million indexable pages had exactly one internal link
pointing to them, and only 2% of the URLs Google had crawled were actually inside the site's real
navigation structure. Most of what little crawl budget the site received was going to URLs outside
the pages that mattered, while the vehicle listing pages that should have driven organic traffic
sat effectively unreachable by a normal crawl path.

Expired vehicle listings compounded the problem from a different angle, generating redirects that
lowered the share of indexable pages and adding more low-value URLs to what Google encountered
when it did crawl the site.

## What changed, and what moved

Botify's team made three categories of change: robots.txt updates to block non-critical
sort-and-filter URL variations, internal linking changes that added homepage links to vehicle
categories and restructured breadcrumbs to follow the Make, Model, City path a real customer
takes, and a cleaned sitemap that dropped expired listings.

Measured over a six-week pilot window, known URLs fell by roughly half, the average number of
inlinks per page rose 15%, and crawl activity to the site's strategic pages rose 19 times over
where it started. Three months out, Botify reports organic search traffic had doubled.

## Why we care

The size of the swing here, a 19x increase in crawl activity, is a function of how badly broken
the starting point was: a 10 million-page site where 99% of pages had never been crawled has far
more room to improve than a typical site already receiving adequate crawl coverage. What
transfers beyond this one engagement is the mechanism Botify names, not the multiplier: pages with
only one internal link, and crawl budget going to URLs outside a site's real structure, are both
concrete, auditable conditions, not vague crawl-budget folklore. A site with either pattern present
at scale has a specific, checkable reason its important pages might be under-crawled, independent
of whatever multiplier its own fix would produce.

## Sources

1. [Crawl Budget Optimization For Classified Websites](https://www.botify.com/blog/crawl-budget-optimization-for-classified-websites) - Botify (primary)
2. [All About Crawl Budget Optimization](https://www.botify.com/blog/crawl-budget-optimization) - Botify