--- title: "REI cut 34 million crawlable URLs to 300,000 by taming facets" url: https://seomadman.com/studies/rei-faceted-navigation-crawl-budget-botify section: studies published: 2026-09-19T00:00:00.000Z modified: 2026-09-19T00:00:00.000Z author: Adam Hafez topics: ["Technical SEO", "Ecommerce SEO"] --- # REI cut 34 million crawlable URLs to 300,000 by taming facets ## The short answer REI's technical SEO team disclosed that its site had 34 million crawlable URLs, driven largely by faceted navigation combinations, before a cleanup cut that figure to 300,000. The team then selectively reintroduced URLs with real search demand. Botify, whose platform REI used to diagnose the problem, published the account from its own customer conference. ## Key takeaways - REI's technical SEO team found the retailer's site had roughly 34 million URLs open to crawling, a scale driven largely by faceted navigation parameter combinations. - The team cut that figure down to 300,000 crawlable URLs by closing off the low-value facet combinations that were consuming crawl budget. - REI reintroduced a smaller set of URLs afterward, limited to facets and combinations with real, demonstrated search demand. - Ryan Ricketts, REI's Technical SEO Manager, presented the account at Botify's Crawl2Convert customer summit rather than in a written case study document. - Botify's own recap of the same story does not publish organic traffic, ranking or revenue figures alongside the URL count, only the crawl-budget outcome. Faceted navigation is the single most common cause Google itself cites for wasted crawling on large sites, and retail category pages are where the damage compounds fastest: every color, size, price band and sort order a shopper can combine has the potential to mint its own crawlable URL. REI's technical SEO team put a number on that damage at Botify's Crawl2Convert customer summit in October 2019, and Botify's own recap of the talk is the source for what follows. ## What REI's team found and what it did Ryan Ricketts, REI's Technical SEO Manager, told the summit that his team's site had roughly 34 million URLs open to crawling. Botify's recap attributes that scale to faceted navigation parameter combinations rather than to REI's actual product catalog, which is a small fraction of that figure. Filter and sort combinations on category pages compound multiplicatively: a handful of facets, each with a handful of values, produces a URL count that has nothing to do with how many distinct products a shopper can actually reach. The team's response was to close that gap down hard: from 34 million crawlable URLs to 300,000. Botify's account does not enumerate REI's exact mechanism, canonical tags, `robots.txt` disallow rules, or `noindex` on low-value combinations, the tools our own [faceted navigation parameter analyzer](/tools/faceted-navigation-parameter-analyzer) is built to help sort between. What the recap does say is that the team then reintroduced a further set of URLs afterward, chosen for having real, demonstrated search demand rather than existing simply because the site's filter logic could generate them. ## What the source does not say This is a crawl-budget story, not a traffic story, and it is worth being precise about that gap. Botify's recap states that REI "saw drastic crawl budget improvements," a qualitative claim it does not attach a percentage, an indexed-page count, or an organic traffic figure to. No ranking data, no revenue figure and no timeframe for the reintroduction phase appear in the source either. Treat the URL count itself, 34 million down to 300,000, as the one hard number this account supports, and treat everything else in it as directional description rather than a measured result. ## Why we care REI's number is a useful gut check for how large a faceted catalog's crawl surface can get before anyone deliberately intervenes: 34 million URLs against a product catalog that could never plausibly be that large is a ratio, not an edge case, and it echoes the pattern Botify separately attributes to Google's Gary Illyes, that faceted navigation alone accounts for half of the crawl issues sites report. The two-step shape of REI's fix, cut hard first, then reintroduce only the combinations with proven demand, is the same order of operations worth applying before touching a single filter on a live category page: block or canonicalize the long tail first, and use actual search-demand data to decide what earns its way back to a crawlable URL. ## Frequently asked questions ### Did REI publish its own case study with these numbers? No. The 34 million to 300,000 figure comes from a talk REI's Ryan Ricketts gave at Botify's Crawl2Convert customer summit in October 2019, which Botify then recapped on its own blog. There is no separate REI-authored case study document behind it. ### Does the source give a traffic or ranking result for REI's cleanup? No. Botify's recap states the URL-count outcome and describes drastic crawl budget improvement in general terms, but it does not publish an organic traffic, ranking or revenue number tied to this specific project. ### What actually drove REI's site to 34 million crawlable URLs? The recap attributes the scale to faceted navigation, the parameter combinations a large retail category page generates when every filter and sort option produces its own crawlable URL, without naming which specific facets REI blocked or canonicalized. ### Did REI block every facet permanently? No. After the initial cut to 300,000 URLs, the team reintroduced a further set of URLs, limited to facet combinations with demonstrated search demand, rather than leaving the site at the reduced count indefinitely. ## Sources 1. [Top 10 Takeaways From Crawl2Convert: Botify's Inaugural Customer Summit](https://www.botify.com/blog/crawl2convert-botifys-inaugural-customer-summit) - Botify (primary)