---
title: "Log analysis cut a jewelry site from 16M to 730K crawlable pages"
url: https://seomadman.com/studies/jetoctopus-log-analysis-parameter-crawl-waste-case-study
section: studies
published: 2026-09-18T00:00:00.000Z
modified: 2026-09-18T00:00:00.000Z
author: Adam Hafez
topics: ["Technical SEO", "Indexing"]
---

# Log analysis cut a jewelry site from 16M to 730K crawlable pages

## The short answer

JetOctopus's case study describes an unnamed jewelry shop whose category, filter, pagination and sort combinations produced about 16 million crawlable pages while relevant pages stayed unindexed for months. After log analysis, the agency recommended closing those combinations, and the site shrank to 730,000 pages, giving Googlebot far fewer URLs to spend its crawl on.

## Key takeaways

- JetOctopus's case study reports a jewelry online shop with over 16 million pages but only 342,000 monthly visits, an extreme mismatch.
- Profitable pages on that site were not indexed for months while Googlebot spent its resources on canonical-tagged filter and pagination URLs.
- The recommendation was to close category, filter, pagination and sort combinations, which cut the site to 730,000 pages.
- The same article shows a real estate site where 23.5 million of 38 million monthly Googlebot requests went to AJAX content.
- The client is unnamed and JetOctopus sells the log analyzer used, so treat this as the vendor's own account, not an audited result.

Server logs record what Googlebot actually requested, not what a crawler simulation guesses. JetOctopus, a log and crawl analysis vendor, published three short case studies from that data in one article. This piece covers the one with the largest measured change, and notes what the article does not say.

## What did the jewelry site's logs show?

The site had more than 16 million pages and 342,000 monthly visits. One part of it ranked at the top of the results, while another part was not indexed for months, and those were relevant, profitable pages rather than outdated ones. JetOctopus crawled 2 million pages, found almost 900 internal links on each, and estimated around 16 million pages in total. It then analyzed the logs and found Googlebot spending resources on pages carrying rel=canonical tags, because the site had placed a canonical or self-canonical on every item in every category, filter and paginated page.

## What was changed and what was the result?

The recommendation was to close the combinations of category, filter, pagination and sort that together generated the bulk of those pages. The source's stated result is that the site's size decreased to 730,000 pages, so the bot could crawl the most profitable pages. The article gives no crawl-request counts, indexed-page counts or traffic figures for the site after the change, and no timeframe. We therefore report only the page-count change.

## Does the article say robots.txt was the fix?

Not for this site. The recommendation says "close" the combinations without naming the mechanism. The article's takeaway does discuss trade-offs: Google recommends noindex over blocking, but on very large e-commerce sites the bot still spends crawl budget fetching a page to read that directive, whereas robots.txt can stop crawling of some sections entirely.

## What do the other two cases in the article show?

On a marketplace with 226,000 pages, filter URLs were disallowed with a `Disallow: /*?` rule, yet duplicates stayed in Google's index. The cause was a conflicting `Allow: /product` rule, and Google follows the least restrictive matching rule. On a real estate site with over 1 million pages, logs showed 38 million Googlebot requests in a month, of which 23.5 million went to AJAX content, and the recommended fix was blocking valueless AJAX in robots.txt. Neither case reports an after figure.

## Sources

1. [How Googlebot Crawls Your Pages. Logs Insights](https://jetoctopus.com/cases/a56-how-googlebot-crawls-your-pages-logs-insights.html) - JetOctopus (primary)