Sitemap vs robots.txt conflict checker

A sitemap tells a search engine “these pages exist, please crawl them.” A robots.txt Disallow rule tells the same crawler “do not fetch this path.” Both files can be published, both can be internally consistent, and still disagree with each other about the same URL. This tool pastes both files in, applies each one’s own real parsing rules, and lists every sitemap URL that lands on the wrong side of that disagreement.

Why this conflict happens in practice#

Nobody sets out to block a URL they also submitted for crawling. It happens because the two files are usually generated by different systems on different schedules: a build step or a CMS plugin regenerates the sitemap automatically from whatever pages exist right now, while robots.txt is a hand-maintained file that someone edited once and rarely revisits. A new section gets added to the disallow list to keep a staging path or an internal search results page out of the index, the sitemap generator has no idea that rule exists, and it keeps listing every URL under that path anyway. The two files drift apart silently because nothing in a normal deploy pipeline checks them against each other, which is exactly the gap this tool fills.

What Google’s documentation confirms, and what it does not#

Google’s robots.txt introduction states plainly that content blocked by robots.txt will not be crawled or indexed by Google, while separately noting that a disallowed URL can still surface in search results without a description if other sites link to it, since the block prevents fetching the page, not knowledge of its existence. Google’s sitemap documentation, read alongside it, never mentions the reverse case: neither page says what happens, or what Search Console reports, when a URL you deliberately submitted in a sitemap is also disallowed. That specific interaction is not addressed in either document, so this tool’s honest claim is limited to the general rule both pages do state, applied mechanically to every URL you paste, rather than any specific documented outcome for the conflict itself.

Using the result#

A flagged conflict is a signal to check intent, not an automatic bug. If the path was disallowed on purpose, the fix is almost always to drop it from the sitemap, since submitting a URL you are simultaneously blocking wastes crawl attention on a page the crawler is refusing to fetch. If the path should be crawlable, the fix is the Disallow rule instead. Either way, treat every conflict this tool reports as a decision to make, and recheck the pair of files here again once you have made it.