--- title: "Noindex does not save crawl budget, per Google's own documentation" url: https://seomadman.com/research/noindex-crawl-budget-reality section: research published: 2026-07-23T00:00:00.000Z modified: 2026-07-23T00:00:00.000Z author: Adam Hafez topics: ["Technical SEO"] --- # Noindex does not save crawl budget, per Google's own documentation ## The short answer Noindex is widely applied as a crawl-budget fix, but Google's own crawl budget documentation says the opposite: Google still requests a noindex page, then drops it after reading the tag, which wastes crawling time rather than saving it. Google's documented alternative, robots.txt, prevents the fetch entirely but can leave an already-indexed page stuck in results indefinitely. ## Key takeaways - Google's large-site crawl budget documentation tells site owners not to use noindex for this purpose, since Google still requests the page and only drops it once it sees the tag. - The same documentation states plainly that this pattern wastes crawling time rather than saving it, which is the opposite of what noindex is commonly deployed to achieve. - Google's robots.txt introduction confirms the documented alternative: content disallowed in robots.txt is not fetched at all, a mechanically different outcome from noindex. - Google's robots.txt documentation states an indexed page can keep appearing, without a description, after a robots.txt block, since Google can no longer crawl it for noindex. - Google recommends robots.txt for pages you do not want crawled at all, reserving noindex for pages that must stay crawlable so Google can act on the tag. Noindex gets reached for constantly as a crawl-budget fix: tag the low-value pages, faceted filters, or thin parameter combinations as noindex, and the thinking goes that Google will stop wasting requests on them. Google's own documentation says this does the opposite of what site owners expect from it. ## The documented mechanism: noindex still costs a crawl Google's [large site owner's guide to managing crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) addresses this pattern directly, not as an aside. Its guidance is not to use noindex for crawl budget management, and it gives the exact reason: Google still requests the page, and only after receiving that response and reading the noindex tag does it drop the page from the index. The crawl already happened by the time the directive takes effect. The documentation is explicit that this wastes crawling time rather than conserving it, which is the precise inversion of why site owners reach for noindex on large sections of a site in the first place. This is a structural property of how the tag works, not a caveat that applies only in some cases. A noindex meta tag lives inside the page's HTML, and a noindex HTTP header lives inside the page's response headers. Either way, Google's crawler has to fetch the full response before either location becomes visible to it. There is no way for Googlebot to know a page carries noindex without first requesting it, which means every noindex page in a crawl budget problem remains exactly as expensive to crawl as it was before the tag was added, indefinitely, since Google has to keep re-requesting it to confirm the tag is still there. The same documentation draws a parallel boundary around robots.txt, though for a different reason: it also advises against using robots.txt to temporarily reallocate crawl budget toward other pages, reserving that mechanism instead for URLs that should not be crawled at all. The two warnings are not the same warning. Noindex fails at crawl budget because the crawl happens anyway. Robots.txt is discouraged for temporary reallocation because it is meant as a durable block, not a budget dial to turn on and off. ## The real alternative, and its own real tradeoff Google's [introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) documents the mechanism that actually withholds the crawl: a URL pattern disallowed in robots.txt is not requested by Google's crawler in the normal case, which is the mechanically different outcome that noindex cannot produce. This is the tool Google's crawl budget guidance points toward for large sections of low-value URLs, faceted navigation and tracking parameters among its own listed examples, precisely because it removes the request instead of letting it happen and discarding the result afterward. That mechanism carries a real, documented cost of its own, and it runs in the opposite direction from the noindex problem. A page already indexed before robots.txt starts blocking it can keep appearing in search results, without a description, because Google no longer crawls that URL to notice anything about it, including a noindex tag placed there afterward or content that no longer exists. Google's own robots.txt documentation is direct about the fix: use noindex, or another block Google can still read, when the actual goal is removing a page from the index, and reach for robots.txt only once that removal has taken effect, or on URLs that were never indexed to begin with. The sequencing matters more than either mechanism alone. A page that needs to come out of the index has to stay crawlable long enough for noindex to be read and acted on; blocking it in robots.txt first removes that path permanently, and a since-deleted or long-stale page can sit in search results with no description for as long as the block stands, because Google has no way to re-check it. The two tools solve different problems, aimed at different points in a page's lifecycle, not two competing fixes for the same one. ## Why we care The crawl-budget fix that gets reached for most often is the one Google's own documentation says does not work for that purpose, and the tool that does work carries a failure mode that looks identical to success from the outside: a page stops showing up as freshly crawled, sits quietly in the index with no description, and nobody notices because nothing is visibly broken. Getting this sequence right on a large site means noindexing pages that need to leave the index while they are still crawlable, waiting for that removal to be confirmed, and only then moving the URL pattern to robots.txt if the goal is to stop the crawl permanently. Reversing that order, or using robots.txt as a shortcut past noindex, produces the exact indefinite-limbo outcome Google's own documentation warns about, on pages that were only ever meant to disappear. ## Frequently asked questions ### Does adding noindex to a page reduce how often Google crawls it? No. Google's own crawl budget documentation states Google still requests the page first and only drops it from the index after reading the noindex tag in the response, which spends crawling time rather than saving it. ### What should be used instead of noindex to actually stop Google from crawling a URL pattern? Google's documentation points to robots.txt for that goal, disallowing the pattern so it is not fetched at all. That is a mechanically different action from noindex, which requires a successful crawl to take effect. ### Is robots.txt a safe replacement for noindex on a page that is already indexed? Not by itself. Google's robots.txt documentation warns that a URL indexed before being disallowed can keep appearing in results, without a description, because Google can no longer crawl it to see a noindex tag or notice it should be dropped. Getting an indexed page out of the index requires noindex, or another block Google can still read, applied before robots.txt cuts off the crawl. ## Sources 1. [Large site owner's guide to managing your crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) - Google Search Central (primary) 2. [Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) - Google Search Central (primary)