Log analysis cut a jewelry site from 16M to 730K crawlable pages

Server logs record what Googlebot actually requested, not what a crawler simulation guesses. JetOctopus, a log and crawl analysis vendor, published three short case studies from that data in one article. This piece covers the one with the largest measured change, and notes what the article does not say.

What did the jewelry site’s logs show?#

The site had more than 16 million pages and 342,000 monthly visits. One part of it ranked at the top of the results, while another part was not indexed for months, and those were relevant, profitable pages rather than outdated ones. JetOctopus crawled 2 million pages, found almost 900 internal links on each, and estimated around 16 million pages in total. It then analyzed the logs and found Googlebot spending resources on pages carrying rel=canonical tags, because the site had placed a canonical or self-canonical on every item in every category, filter and paginated page.

What was changed and what was the result?#

The recommendation was to close the combinations of category, filter, pagination and sort that together generated the bulk of those pages. The source’s stated result is that the site’s size decreased to 730,000 pages, so the bot could crawl the most profitable pages. The article gives no crawl-request counts, indexed-page counts or traffic figures for the site after the change, and no timeframe. We therefore report only the page-count change.

Does the article say robots.txt was the fix?#

Not for this site. The recommendation says “close” the combinations without naming the mechanism. The article’s takeaway does discuss trade-offs: Google recommends noindex over blocking, but on very large e-commerce sites the bot still spends crawl budget fetching a page to read that directive, whereas robots.txt can stop crawling of some sections entirely.

What do the other two cases in the article show?#

On a marketplace with 226,000 pages, filter URLs were disallowed with a Disallow: /*? rule, yet duplicates stayed in Google’s index. The cause was a conflicting Allow: /product rule, and Google follows the least restrictive matching rule. On a real estate site with over 1 million pages, logs showed 38 million Googlebot requests in a month, of which 23.5 million went to AJAX content, and the recommended fix was blocking valueless AJAX in robots.txt. Neither case reports an after figure.