REI cut 34 million crawlable URLs to 300,000 by taming facets

Faceted navigation is the single most common cause Google itself cites for wasted crawling on large sites, and retail category pages are where the damage compounds fastest: every color, size, price band and sort order a shopper can combine has the potential to mint its own crawlable URL. REI’s technical SEO team put a number on that damage at Botify’s Crawl2Convert customer summit in October 2019, and Botify’s own recap of the talk is the source for what follows.

What REI’s team found and what it did#

Ryan Ricketts, REI’s Technical SEO Manager, told the summit that his team’s site had roughly 34 million URLs open to crawling. Botify’s recap attributes that scale to faceted navigation parameter combinations rather than to REI’s actual product catalog, which is a small fraction of that figure. Filter and sort combinations on category pages compound multiplicatively: a handful of facets, each with a handful of values, produces a URL count that has nothing to do with how many distinct products a shopper can actually reach.

The team’s response was to close that gap down hard: from 34 million crawlable URLs to 300,000. Botify’s account does not enumerate REI’s exact mechanism, canonical tags, robots.txt disallow rules, or noindex on low-value combinations, the tools our own faceted navigation parameter analyzer is built to help sort between. What the recap does say is that the team then reintroduced a further set of URLs afterward, chosen for having real, demonstrated search demand rather than existing simply because the site’s filter logic could generate them.

What the source does not say#

This is a crawl-budget story, not a traffic story, and it is worth being precise about that gap. Botify’s recap states that REI “saw drastic crawl budget improvements,” a qualitative claim it does not attach a percentage, an indexed-page count, or an organic traffic figure to. No ranking data, no revenue figure and no timeframe for the reintroduction phase appear in the source either. Treat the URL count itself, 34 million down to 300,000, as the one hard number this account supports, and treat everything else in it as directional description rather than a measured result.

Why we care#

REI’s number is a useful gut check for how large a faceted catalog’s crawl surface can get before anyone deliberately intervenes: 34 million URLs against a product catalog that could never plausibly be that large is a ratio, not an edge case, and it echoes the pattern Botify separately attributes to Google’s Gary Illyes, that faceted navigation alone accounts for half of the crawl issues sites report. The two-step shape of REI’s fix, cut hard first, then reintroduce only the combinations with proven demand, is the same order of operations worth applying before touching a single filter on a live category page: block or canonicalize the long tail first, and use actual search-demand data to decide what earns its way back to a crawlable URL.