Mueller: why a large aggregator is crawled but barely indexed
John Mueller told a site owner that URLs indexed from a large product aggregator redirect to the homepage, and that such sites are hard to justify.
Technical SEO is the work of making a site reliably crawlable, renderable and indexable: correct status codes, canonical signals, sitemaps, robots directives, fast responses and stable URLs. It does not make weak content rank, but its absence reliably prevents strong content from ranking.
Reference: developers.google.com
John Mueller told a site owner that URLs indexed from a large product aggregator redirect to the homepage, and that such sites are hard to justify.
Google's John Mueller says AI crawlers can't submit sitemaps and that Couldn't fetch on a valid sitemap often comes down to host load or crawl demand.
All 23 typical and slowest times Gary Illyes showed at Search Central Live Barcelona, as recapped by ROAST, plus caveats and Google's own docs.
RSS feed validator: paste feed XML to check RSS 2.0 and Atom 1.0 for parse errors, required elements, dates, guids, enclosures and URLs. Runs in-browser.
Google Ads updated its business name requirements policy, permitting name differences from destination domains for verified brands while barring resellers.
Google's partner-only Web Search Service API docs were refreshed in September and now return a 404. An archived copy shows them. Google has not commented.
SearchPilot split-tested turning unlinked brand text on product detail pages into links to brand category pages, finding an 18% uplift in organic traffic.
humans.txt generator: build a humans.txt file with TEAM, THANKS and SITE sections from the humanstxt.org template. Free, runs in your browser.
Big Daddy was Google's 2005 to 2006 crawling and indexing upgrade. Matt Cutts named it and said it was fully deployed by 29 March 2006.
Bourbon was a staged 2005 Google update that a Google employee described in forum posts as 3.5 improvements in search quality.
Google announced Caffeine complete on 8 June 2010: a new web index Google said gave 50 percent fresher results. It was an indexing change.
Google's exact match domain update of September 2012 reduced low-quality exact-match domains in results, affecting 0.6% of English-US queries.
Google's page experience update for desktop ran 9 days, 22 February to 3 March 2022, bringing mobile's page experience signals to desktop ranking.
On 7 August 2014 Google made HTTPS a lightweight ranking signal, affecting fewer than 1% of global queries. What it changed and what applies today.
From 10 January 2017, Google said mobile pages whose content is hidden behind intrusive interstitials may not rank as high. Here is what applies today.
Google's June 2021 spam update ran in two one-day parts, on 23 and 28 June 2021. What Google confirmed about it and what its spam policies say today.
Google's mobile-friendly update began on 21 April 2015, boosting mobile-friendly pages in mobile results. The industry called it Mobilegeddon.
Google strengthened its mobile-friendly ranking signal in May 2016, fully rolled out by 12 May. Some in the industry called it Mobilegeddon 2.
Google's October 2022 spam update ran two days, 19 to 21 October 2022, in all languages. Dates, what Google said and how to review your site.
Google's October 2023 spam update ran from 4 to 19 October 2023 and targeted cloaking, hacked, auto-generated and scraped spam in many languages.
Google's page experience update rolled out on mobile from 15 June to 2 September 2021, adding Core Web Vitals to ranking. What it means today.
Google's page layout algorithm of 19 January 2012 demoted pages with too many ads above the fold. It affected less than 1% of searches, Google said.
Bing is testing a sitelinks variant that reveals a description under a sitelink only when the cursor hovers over it, per a 24 September 2026 report.
Statcounter puts Google at 91.02% of worldwide search in August 2026, ahead of Bing at 4.55%, against 90.37% across the prior twelve months.
The 2025 Web Almanac puts the median home page at 2.86 MB on desktop and 2.56 MB on mobile, with images and JavaScript the two largest parts.
John Mueller said on record that Google does not use Analytics bounce rate for ranking. What that statement covers, and what it leaves open.
HTTP Archive's 2024 Web Almanac SEO chapter compared mobile and desktop crawls. Titles, canonicals and word counts nearly match; a few gaps remain.
Ahrefs checked Mention.com's January 2016 move from subdomain to subfolder in Site Explorer. Traffic shifted, and only new content produced growth.
searchviu's client case: NFON merged nfon.at into nfon.com/at/ and saw Austrian organic traffic up 90% in five months, with a Germany dip.
Cloudflare Radar's 2025 Year in Review puts mobile at 43% of global requests, up from 41%, with mobile ahead in 117 countries and regions.
Image lazy loading checker: paste HTML to test every img tag for width and height, lazy loading on the likely LCP image. Free, runs in your browser.
Google now writes region codes in uppercase in its hreflang examples, such as en-GB, but its documentation says case does not matter to Google.
Google's Page Experience documentation now lists experimental CrUX ad metrics: ad count, ad density and ad weight. It does not call them a ranking signal.
NewzDash's analysis of 400+ news publishers shows Discover's share of Google traffic rising from 37.03% in 2023 to 67.51% in Q4 2025.
HTTP Archive's 2025 Web Almanac measured HTTPS on 97.3% of mobile homepages and 98.8% of requests. What the crawl method can and cannot show.
Ahrefs checked the age of top 10 pages for 1.3 million US keywords: 72.9 percent are over three years old, and the average number one page is five.
Backlinko's 11.8M-result study found shorter URLs correlate with slightly better Google rankings. Here is the method, the numbers and the limits.
On's Prerender.io case study reports +67% Google traffic and +98% Bing traffic after prerendering a JavaScript front end instead of building SSR.
Google added a requirements section to Merchant Center's Store ratings doc: a 100-review minimum, a 3.5-star floor, a 24-month window, per-country scoring.
Black hat SEOs published spam through public Gemini NotebookLM notebooks and Google indexed over 12,000 pages; it removed the directory the next day.
Google's case study on Italiaonline reports an 841% rise in video clicks and 353% more impressions after fixing VideoObject structured data errors.
Noindex, canonical and sitemap conflict checker: see where one page's three signals contradict Google's own guidance. Free, runs in your browser.
Backlinko and Ahrefs analyzed 11.8 million search results and found no correlation between page HTML size and Google rankings.
Skalum's case study on Shopify brand Silly Santa: hreflang across 7 ccTLDs, with UK organic users up 297.5% and Norway's local traffic share up to 93%.
Croton Content's audit found 324,580 broken URLs after a rebrand with no redirects. Fixing them took weekly organic sessions from 974 to a 12,276 peak.
Open Graph tag checker: tests pasted og meta tags for the four required properties, relative URLs, duplicates and og:type. Free, runs in your browser.
Since around September 13, 2026, rank-tracking and tracking tools report Google blocking more of their scraping, Nozzle citing an 80 percent data drop.
Portent's aggregate analysis of 100 million page views across 20 sites found B2C conversion falls from 3.05% at 1 second to 0.67% at 4 seconds.
REI's technical SEO team told Botify's Crawl2Convert summit how it cut its site from 34 million URLs to 300,000, including faceted navigation.
Meta viewport tag validator: checks the content string for unknown keys, width=device-width and zoom blocking under WCAG 1.4.4. Free, runs in your browser.
Search Engine Roundtable reports Bing showing a verify-you-are-human checkbox before results. The trigger is unknown and the check was not reproduced.
Google cut the follower requirement for Search profiles to 10,000, added richer article views, and lets one account manage up to 10 profiles.
The 2025 Web Almanac SEO chapter puts rel=canonical on 68% of desktop and 67% of mobile pages, up from 65%, with mismatches under 1%.
Cloudflare reports AI training made up 52% of identified crawler requests in June 2026, up from 22% in Spring 2025, with search crawling shrinking.
The 2025 Web Almanac's INP data: 77% of mobile sites have good INP, popular sites trail at 63%, and mobile home pages beat inner pages 80% to 69%.
Bing's study found 18% of sitemaps with lastmod set it wrongly, usually to the generation date, and Google says it uses only accurate ones.
A 36-method academic benchmark found paragraph-based chunking nearly doubles retrieval accuracy over naive fixed-size splitting across six domains.
Sistrix analyzed 80 million keywords and found position 1 averages 28.5% CTR, ranging from 13.7% to 46.9% depending on the SERP layout.
Uproer's published case study reports feed-level Google Merchant Center changes lifted clicks 130%, 116% and 3,500 for three unnamed ecommerce brands.
A JetOctopus case study: log files showed Googlebot spending crawl on filter, sort and pagination URLs, and closing them shrank the site to 730K pages.
Nuvemshop's web.dev case study: removing lazy loading on top images and adding fetchpriority took good LCP from 57% to 96% across its stores.
Canonical chain and loop checker: paste url -> canonical pairs, follow every hop and catch loops and multi-hop chains. Free, runs in your browser.
Hreflang sitemap generator: turn language codes and URLs into sitemap XML with xhtml:link alternates, and check codes and URLs. Free, runs in your browser.
HTTP status code lookup: see how Google's crawlers handle each code you paste, plus a redirect hop count against the 10-hop limit. Free, runs in browser.
Nofollow link checker for pasted HTML: audits every link's rel against Google's sponsored, ugc and nofollow rules. Free, runs in your browser.
Robots meta tag validator: check meta robots and X-Robots-Tag rules against Google's list, and catch bad values and conflicts. Free, runs in your browser.
URL normalizer: paste a list of URLs, normalize them by RFC 3986 and your site policy, and find duplicate variants of one page. Free, runs in your browser.
GA4 Dashboards let you drag dimensions and metrics onto a grid canvas, combining six chart types into one KPI page instead of separate reports.
Google's Search Central team holds a free, one-day Search Central Live event in Bengaluru on October 30, 2026, its first India event since 2023.
The 2025 Web Almanac's CMS chapter breaks Core Web Vitals pass rates down by platform: Duda at 85%, Wix up from 55% to 74%, WordPress at 45%.
The 2024 Web Almanac breaks JSON-LD down by schema.org type: WebSite tops out at 12.73% of mobile pages, Organization at 7.16%, Product at just 0.77%.
Semrush Sensor's live index reads 2.1 out of 10 for mid-September 2026, but News and Finance run far hotter than Science or Health.
A SearchPilot split test added 301 redirects atop an existing canonical tag for trailing-slash duplicates. The result stayed inconclusive.
Botify's case study of a US auto marketplace: 99% of pages uncrawled, then 19x more crawl activity after linking, robots.txt and sitemap fixes.
Ahrefs found a 33.4% title rewrite rate, far below Zyppy's 61.6% and Search Engine Land's 76%. The gap traces to how each study defines a match.
DebugBear's case study: CoinStats updated about 15,000 pages and moved off Base64 images; it reports Good URLs and their impressions up 3x.
Uproer's case study on IFTTT: searchbots reached only 59 pages via internal links until a fix, after which the site saw 33% YoY organic traffic growth.
Inflow's case study: Simms Fishing saw organic traffic up 126% YoY and non-brand category traffic up 5,642% after a Magento to Shopify move.
Faceted navigation URL checker: test a filtered URL's parameters against Google's crawling guidance and get a robots.txt block. Free, runs in your browser.
ccTLD vs subdomain vs subdirectory: rank four international URL structures against Google's documented pros and cons for you. Free, runs in your browser.
Meta refresh redirect checker: parses the delay and target, then shows whether Google reads it as permanent or temporary. Free, runs in your browser.
Pagination canonical tag checker: catch later pages canonicalized to page 1, repeated URLs and page numbers after a #. Free, runs in your browser.
Robots.txt validator: lint every line for syntax errors, unsupported fields, bad paths and relative sitemap URLs. Free, runs in your browser.
Google's September 2026 changelog documents aggregator units, supplier units and carousels: result formats limited to the EEA, South Africa and Turkiye.
We read the robots.txt of 60 publishers and SEO vendors. News sites block AI crawlers almost universally. The SEO industry does not.
Only 14 of 60 major sites serve an llms.txt. Half of them sell SEO or hosting software. One is a news publisher. The file has an audience problem.
Redirect chain checker: paste the hops you collected and check hop count, loops, mixed 301 and 302 codes and https drops. Free, runs in your browser.
Image alt text checker: paste HTML to find missing, empty, generic and keyword-stuffed alt attributes on every img tag. Free, runs in your browser.
robots.txt and llms.txt generator for AI crawlers: allow or block GPTBot, ClaudeBot, Google-Extended and 11 more by purpose. Free, runs in your browser.
Schema markup generator: build JSON-LD for Article, NewsArticle, FAQPage, VideoObject, Organization and BreadcrumbList. Free, runs in your browser.
Meta title and description checker: measure title and meta description length in pixels and characters with a Google preview. Free, runs in your browser.
Google's August 2026 spam update ran from 18 to 21 August. Here is how to tell a spam-update drop from a core-update drop in your own data.
Google's Search Central team adds Bogota to its live event roster and returns to Ciudad de Mexico in October, alongside a three-day Deep Dive in Barcelona.
Two independent studies measured how often Google rewrites a title tag rather than using it as written: 61.6% in 2022, 76% by 2025.
Redirect map generator for site migrations: map old URLs to new and export CSV, .htaccess Redirect 301 or Nginx rewrite rules. Free, runs in your browser.
Two separate September 2026 Bing tests change result snippets: one bolds the https segment of the URL, the other drops the site name line.
Core Web Vitals checker: rate LCP, INP and CLS values as Good, Needs improvement or Poor and see if all three pass. Free, runs in your browser.
SearchPilot split-tested removing a self-referential breadcrumb link on PLPs. Traffic fell 5.5%, likely from a schema markup mismatch left behind.
Canonical tag checker: test a page URL and its rel=canonical for absolute URL, HTTPS, self or cross reference, domain and tracking. Free, in your browser.
Robots.txt tester: check if a URL path is blocked for Googlebot, GPTBot or any bot, and see the exact rule that decides it. Free, runs in your browser.
Sitemap vs robots.txt checker: find sitemap URLs that a robots.txt Disallow rule blocks for Googlebot or any user-agent. Free, runs in your browser.
SearchPilot canonicalized product pages to a variant's URL instead of self-referencing. Variant pages gained 22%; main pages stayed inconclusive.
Bulk meta title and description length checker: paste many pages and see which titles and descriptions truncate, in pixels. Free, runs in your browser.
From the May 2026 spam-policy revision to the August spam update, tracing enforcement against mass-produced, low-value content across 2026.
Google has enforced its site reputation abuse policy since May 2024. Here is what is documented versus what is anecdotal case tracking.
JetOctopus's case study on real estate site DOM.RIA, reworking links to poorly-crawled directory pages more than doubled Googlebot visits.
Hreflang tag generator: build link rel=alternate hreflang tags for localized URLs and check codes, self-references and x-default. Free, runs in browser.
Google fixed a bug replacing site favicons with a default globe icon in search results, and separately restored a two-day gap in Crawl Stats data.
Five independent trackers measured llms.txt adoption since late 2025 and reported 7.4% to 28%, because none counted the same thing.
SEO-friendly URL checker: test a URL's separators, parameters, length, depth and opaque IDs against Google's URL guidance. Free, runs in your browser.
XML sitemap validator: check sitemap XML for parse errors, namespace, loc, lastmod, changefreq, priority and URL limits. Free, runs in your browser.
A 892-migration study found the average new domain took 523 days to match the old domain's organic traffic, and 17% never did within 1,000 days.
Meta robots tag generator: build noindex, nofollow, max-snippet and other rules as a meta tag or X-Robots-Tag header. Free, runs in your browser.
The IETF's AI Preferences group is standardizing AI training and search signals that extend robots.txt. Still an Internet-Draft, not a shipped standard.
Google's web.dev logged two named A/B tests: Rakuten 24 saw 53.37% more revenue per visitor, Vodafone saw 8% more sales after a 31% LCP gain.
Google search operators builder: combine site:, filetype:, quotes, minus, intitle: and OR into one query and open it in Google. Free, runs in your browser.
Zyppy's 23-million-link study and Google's own crawling documentation show why pages with few internal links struggle to get discovered and clicked.
SearchPilot split-tested changing a travel site's top heading from an H2 to an H1. The change produced a statistically significant 4.5% traffic increase.
HTTP Archive's Web Almanac found JSON-LD on 41% of pages in 2024, up from 34% in 2022, while RDFa and Open Graph stayed the most common formats.
Deloitte's 2019 study of 37 brand sites found a 0.1s mobile speed gain lifted retail conversion 8.4%. Shopify data adds a 100ms benchmark.
HTTP Archive's July 2025 data broken down by metric: INP splits mobile and desktop by 20 points, LCP by 12, and CLS reverses the pattern entirely.
rel=next/prev has been dead as an indexing signal since 2019, per Google's own words. Infinite scroll without real URLs is the same mistake in new clothes.
Two widely repeated hreflang error statistics do not trace to any real source. Here is what could be verified instead, from Google's own documentation.
SearchPilot split-tested linking each of 8,000 regional pages to its 6 nearest neighbors. Linked pages saw a 7% organic traffic uplift over controls.
A common crawl-budget fix is backwards. Google's own documentation says noindex still costs a full request before the page gets dropped from the index.
Court testimony and a leaked Google document, read together, describe a click-based ranking system Google spent years declining to confirm in public.
CrUX data and a 20,000-URL ranking study both show a link between Core Web Vitals and rank, but Google's own docs put content relevance first.
A 2024 Vercel and MERJ study measured the real gap between crawl and render at scale. The median was seconds; the tail stretched to a full day.
Google publishes a real, maintained list of the named systems behind Search ranking. Read directly, not summarized secondhand.
Three trackers measured 2026 organic CTR by position and reported position 1 anywhere from 27.6% to 39.8%, because none used the same method.
Tracing Google's mobile-first indexing announcements from 2016 to the 2023 completion notice, and its documentation on mobile-desktop content parity.
x-default has been explained by Google at least three times since 2013. Here is the real, dated wording each time, and what current documentation says.
Google called HTTPS a very lightweight ranking signal in 2014 and has never revised that number. Adoption data now shows why it barely needs to.
Cloudflare's CEO says automated requests for HTML pages now outnumber human ones, ahead of his own 2027 forecast.
Google has never published a percentage for how much link equity a 301 redirect passes. The 90-99% figure in circulation comes from somewhere else.
Featured snippet prevalence and CTR figures vary by nearly 2x across sources. Here is which trace to a real, dated study, and which don't trace anywhere.
FAQPage markup no longer earns an expanded snippet in Google Search as of 7 May 2026, though the markup stays valid and Google says leaving it is harmless.
Portent, Ahrefs and Search Engine Journal put Google's meta description rewrite rate at 71%, 62.78% and over 70% in 2020. Only two are real studies.
Bing's own published numbers show IndexNow submissions climbing steadily since 2022, yet Google still has no roadmap to join the instant-indexing protocol.
Google's own dated posts trace the page experience rollout from 2020 to its 2022 desktop finish, and what the 2026 docs now drop.
Google's May 2026 broad core update ran 21 May to 2 June, its second confirmed core update of 2026. What the dates mean for reading your own traffic.
Independent.ie gained 55% mobile traffic on AMP. Kinsta lost 59% of its leads. Here is what really happened across five documented, named case studies.
Schema.org's own release notes put the vocabulary at 823 types and 1,529 properties as of version 30.0, alongside a new Google-backed usage dataset.
Google's own Maps blog states 2025 review-fraud enforcement scale, and its April 2026 update bans staff review quotas and naming employees in reviews.
SearchPilot split-tested moving breadcrumbs to the top and bottom of listing pages. Both tests were inconclusive, showing placement does not alter rank.
SearchPilot split-tested full and partial capitalization in title tags on listing pages, finding a 17.5% organic uplift led by mobile search gains.