Mueller: AI crawlers read sitemaps, Couldn't fetch explained

Google’s John Mueller and Martin Splitt spent the 1 October 2026 episode of Search Off the Record, Do sitemaps still matter?, on two sitemap questions. Search Engine Journal reported what Mueller said in one article by Matt G. Southern and one by Roger Montti. We did not hear the episode: its YouTube page confirms the title and the 1 October upload, and every statement below is as reported by those two writers.

Can AI crawlers find your sitemap?#

Per Southern, Mueller said AI training crawlers usually have no console or setup where a site owner can submit a sitemap file. For sites that want their content in AI systems, he suggested either keeping the generic name sitemap.xml or focusing on RSS feeds. Feeds are easier to find, he noted, because pages usually link them from the HTML head.

He also said he has seen an AI crawler access his sitemap in his server logs, and the same with his RSS files. He did not name the crawlers, and said he does not know whether AI companies document this or what they do with the files.

Southern adds that the sitemaps protocol treats the robots.txt Sitemap line as independent of the user-agent line, so it is not tied to one crawler’s rules. Check the pair with the sitemap vs robots.txt checker.

What about a private sitemap?#

Mueller said a site owner who wants a sitemap kept private can give it an unusual file name, leave it out of robots.txt and submit it to Google directly. The downside, per Southern’s report, is that other systems cannot find it, and Bing would probably need its own submission.

Does llms.txt replace a sitemap?#

Asked that, Mueller compared the Markdown file to an HTML sitemap and said Google’s systems cannot use it as a sitemap because it lacks the strict format. Southern reports he said the hope is bigger than the reality, that none of this happens today, and that he would not rely on it. See the W3C llms.txt draft for where the format stands.

Why do valid sitemaps fail to fetch?#

Splitt put the question: the XML validates, it is public and robots.txt links it, yet Search Console sometimes says it can’t fetch it. Per Montti, Splitt acknowledged this happens to many people with valid sitemaps. Mueller gave two reasons:

• Host load: Google’s systems may be too busy with other crawling to fetch the sitemap, and Search Console reports that as Couldn’t fetch too.
• Crawl demand: if Google’s systems see no need to crawl much more from a site, they may skip the sitemap. Mueller said crawl demand is very often based on perceived quality, so it is not purely a technical thing.

Montti reads a third point into Mueller’s answer: Google may simply not need the sitemap, and would use it if site quality improved significantly. Montti’s own view is that the message does not match the actual reason, which leaves Search Console users confused, and that Google knows this and has done nothing. That is his opinion, not Google’s.

What should you rule out first?#

Google’s Sitemaps report help page lists low crawl demand among the reasons a fetch fails, and says the higher the quality of the site’s content, the higher the crawl demand. It also lists a robots.txt block, an unresolved manual action and a wrong URL that returns a 404.

Those are the checks you can run: confirm the file parses with the XML sitemap validator, then look in your server logs for failed requests around the fetch time. Southern points to logs as where Mueller saw AI crawlers, and where you can check your own. If everything is clean, the reporting points to demand and quality, not the file.