Noindex does not save crawl budget, per Google's own documentation

Noindex gets reached for constantly as a crawl-budget fix: tag the low-value pages, faceted filters, or thin parameter combinations as noindex, and the thinking goes that Google will stop wasting requests on them. Google’s own documentation says this does the opposite of what site owners expect from it.

The documented mechanism: noindex still costs a crawl#

Google’s large site owner’s guide to managing crawl budget addresses this pattern directly, not as an aside. Its guidance is not to use noindex for crawl budget management, and it gives the exact reason: Google still requests the page, and only after receiving that response and reading the noindex tag does it drop the page from the index. The crawl already happened by the time the directive takes effect. The documentation is explicit that this wastes crawling time rather than conserving it, which is the precise inversion of why site owners reach for noindex on large sections of a site in the first place.

This is a structural property of how the tag works, not a caveat that applies only in some cases. A noindex meta tag lives inside the page’s HTML, and a noindex HTTP header lives inside the page’s response headers. Either way, Google’s crawler has to fetch the full response before either location becomes visible to it. There is no way for Googlebot to know a page carries noindex without first requesting it, which means every noindex page in a crawl budget problem remains exactly as expensive to crawl as it was before the tag was added, indefinitely, since Google has to keep re-requesting it to confirm the tag is still there.

The same documentation draws a parallel boundary around robots.txt, though for a different reason: it also advises against using robots.txt to temporarily reallocate crawl budget toward other pages, reserving that mechanism instead for URLs that should not be crawled at all. The two warnings are not the same warning. Noindex fails at crawl budget because the crawl happens anyway. Robots.txt is discouraged for temporary reallocation because it is meant as a durable block, not a budget dial to turn on and off.

The real alternative, and its own real tradeoff#

Google’s introduction to robots.txt documents the mechanism that actually withholds the crawl: a URL pattern disallowed in robots.txt is not requested by Google’s crawler in the normal case, which is the mechanically different outcome that noindex cannot produce. This is the tool Google’s crawl budget guidance points toward for large sections of low-value URLs, faceted navigation and tracking parameters among its own listed examples, precisely because it removes the request instead of letting it happen and discarding the result afterward.

That mechanism carries a real, documented cost of its own, and it runs in the opposite direction from the noindex problem. A page already indexed before robots.txt starts blocking it can keep appearing in search results, without a description, because Google no longer crawls that URL to notice anything about it, including a noindex tag placed there afterward or content that no longer exists. Google’s own robots.txt documentation is direct about the fix: use noindex, or another block Google can still read, when the actual goal is removing a page from the index, and reach for robots.txt only once that removal has taken effect, or on URLs that were never indexed to begin with.

The sequencing matters more than either mechanism alone. A page that needs to come out of the index has to stay crawlable long enough for noindex to be read and acted on; blocking it in robots.txt first removes that path permanently, and a since-deleted or long-stale page can sit in search results with no description for as long as the block stands, because Google has no way to re-check it. The two tools solve different problems, aimed at different points in a page’s lifecycle, not two competing fixes for the same one.

Why we care#

The crawl-budget fix that gets reached for most often is the one Google’s own documentation says does not work for that purpose, and the tool that does work carries a failure mode that looks identical to success from the outside: a page stops showing up as freshly crawled, sits quietly in the index with no description, and nobody notices because nothing is visibly broken. Getting this sequence right on a large site means noindexing pages that need to leave the index while they are still crawlable, waiting for that removal to be confirmed, and only then moving the URL pattern to robots.txt if the goal is to stop the crawl permanently. Reversing that order, or using robots.txt as a shortcut past noindex, produces the exact indefinite-limbo outcome Google’s own documentation warns about, on pages that were only ever meant to disappear.