News
Google adds UGC Fresh Data Program for large platforms
Google's UGC Fresh Data Program lets approved platforms push user-generated content to Search outside crawling and the Indexing API.
Indexing - getting pages discovered and kept
What Indexing means
Indexing is the stage between crawling and ranking where a search engine decides whether to store a page and under which canonical URL. A page can be crawled successfully and still never be indexed, usually because it duplicates another URL or is judged too thin to be worth storing.
News
Mueller: why a large aggregator is crawled but barely indexed
John Mueller told a site owner that URLs indexed from a large product aggregator redirect to the homepage, and that such sites are hard to justify.
News
Mueller: AI crawlers read sitemaps, Couldn't fetch explained
Google's John Mueller says AI crawlers can't submit sitemaps and that Couldn't fetch on a valid sitemap often comes down to host load or crawl demand.
Reports
How long Google takes to crawl, index and update results
All 23 typical and slowest times Gary Illyes showed at Search Central Live Barcelona, as recapped by ROAST, plus caveats and Google's own docs.
Tools
RSS feed validator: check RSS 2.0 and Atom feeds
RSS feed validator: paste feed XML to check RSS 2.0 and Atom 1.0 for parse errors, required elements, dates, guids, enclosures and URLs. Runs in-browser.
Research
Mobile vs desktop parity in the 2024 Web Almanac SEO chapter
HTTP Archive's 2024 Web Almanac SEO chapter compared mobile and desktop crawls. Titles, canonicals and word counts nearly match; a few gaps remain.
Studies
Brafton's accidental sitewide noindex: a six-week recovery
A sprint pushed noindex onto every Brafton page in August 2019. Search traffic fell 33.2% at the peak and took six weeks to return to baseline.
Tools
Sitemap lastmod checker: W3C Datetime format audit
Sitemap lastmod checker: audit every lastmod in an XML sitemap against W3C Datetime and flag future and identical dates. Free, runs in your browser.
Tools
X-Robots-Tag header checker and validator
X-Robots-Tag header checker: validate rules, user-agent prefixes, max-snippet and preview values, dates and conflicts. Free, runs in your browser.
News
Spammers are getting NotebookLM public pages indexed by Google
Black hat SEOs published spam through public Gemini NotebookLM notebooks and Google indexed over 12,000 pages; it removed the directory the next day.
Tools
Noindex, canonical and sitemap conflict checker
Noindex, canonical and sitemap conflict checker: see where one page's three signals contradict Google's own guidance. Free, runs in your browser.
Reports
Canonical tag adoption, 2025: 68% of desktop pages set one
The 2025 Web Almanac SEO chapter puts rel=canonical on 68% of desktop and 67% of mobile pages, up from 65%, with mismatches under 1%.
Research
How accurate are sitemap lastmod values? Bing's study
Bing's study found 18% of sitemaps with lastmod set it wrongly, usually to the generation date, and Google says it uses only accurate ones.
Studies
Log analysis cut a jewelry site from 16M to 730K crawlable pages
A JetOctopus case study: log files showed Googlebot spending crawl on filter, sort and pagination URLs, and closing them shrank the site to 730K pages.
Tools
Hreflang XML sitemap generator
Hreflang sitemap generator: turn language codes and URLs into sitemap XML with xhtml:link alternates, and check codes and URLs. Free, runs in your browser.
Tools
HTTP status code lookup for Google crawling
HTTP status code lookup: see how Google's crawlers handle each code you paste, plus a redirect hop count against the 10-hop limit. Free, runs in browser.
Tools
Robots meta tag validator with combined Googlebot result
Robots meta tag validator: check meta robots and X-Robots-Tag rules against Google's list, and catch bad values and conflicts. Free, runs in your browser.
Tools
URL normalizer: find duplicate URL variants
URL normalizer: paste a list of URLs, normalize them by RFC 3986 and your site policy, and find duplicate variants of one page. Free, runs in your browser.
Research
Who blocks AI crawlers: 60 robots.txt files, read in full
We read the robots.txt of 60 publishers and SEO vendors. News sites block AI crawlers almost universally. The SEO industry does not.
News
Publisher reports Google News sitemap data stuck in Search Console
A publisher says the indexed pages report for its news sitemap has not moved since August 28, even though new articles are indexed and visible.
Tools
Hreflang tag generator and checker
Hreflang tag generator: build link rel=alternate hreflang tags for localized URLs and check codes, self-references and x-default. Free, runs in browser.
Reports
Mobile-first indexing, from 2016 test to 2023 completion
Tracing Google's mobile-first indexing announcements from 2016 to the 2023 completion notice, and its documentation on mobile-desktop content parity.