Mueller: why a large aggregator is crawled but barely indexed

A site owner asked why Google would not index even a homepage that Googlebot had crawled heavily. John Mueller answered on Bluesky that the site had technical problems and was a hard kind of site to make worthwhile. We could not open his post, so the wording below is as Search Engine Roundtable quotes it on 6 October 2026.

What did Mueller say?#

Per the report, the question was that Googlebot had crawled the site 1.17M times in the last few weeks, yet Google would not index the homepage, and Search Console showed 4 indexed URLs in total. Search Engine Roundtable adds that the page indexing report listed four pages indexed out of 180,000, and that the crawl stats chart showed a spike.

Mueller called it a “large product aggregator” and named two issues. The first is the one he quoted directly: “a bunch of the URLs that are indexed from your site, they all redirect to the homepage, so it seems like there are still some big technical issues that you need to resolve on your site.”

Why do the redirects matter?#

The report gives no further detail on the redirects, and we will not guess at the site’s setup. What Mueller’s wording does say is that the URLs which made it into the index redirect to the homepage instead of resolving to pages of their own.

If you run a catalog, you can test this on your side. Run the URLs your report lists as indexed through the redirect chain analyzer, and compare near-duplicate variants with the URL normalizer and duplicate variant finder. Both checks are our suggestion, not something Mueller prescribed.

Why is a large aggregator hard?#

Mueller’s second point was general. Per the report, another large product aggregator like this is not trivial to make and maintain well. You end up with a ton of URLs, apparently with programmatic, autogenerated content, and convincing search engines and users that this is worthwhile will be hard.

Search Engine Roundtable adds that Google is generally not a fan of programmatic SEO. That is Search Engine Roundtable’s reading rather than a statement from Mueller. Our report on the spam updates covers how Google treats scaled content. If your catalog also produces filter and sort URLs, the faceted navigation parameter analyzer shows how many variants you expose.

What should you take from it?#

Crawl volume is not an indexing signal. In the case reported, a crawl spike sat next to four indexed pages, so a busy crawl stats chart tells you Google fetched URLs, not that it kept them. Start with what the indexing report says is indexed, and fix any URL that redirects away before judging content quality. The wider checklist sits under technical SEO.