Skip to content
SEO Madmanby Adam Hafez
Free tool

Robots.txt syntax validator

Lint a whole robots.txt file for structural and syntax errors against RFC 9309 and Google's own interpretation of it, line by line.

Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.

Nothing you paste here is sent anywhere; it is parsed line by line in your browser only.

    Key takeaways

    • A valid robots.txt line is a field, a colon, and a value; a line with no colon is invalid and gets ignored.
    • Google's documentation supports exactly four fields: user-agent, allow, disallow and sitemap, and states other fields such as crawl-delay are not supported.
    • An allow or disallow path value must start with a slash to designate the root, per Google's documentation.
    • A sitemap value must be a fully qualified, absolute URL, not a path relative to the robots.txt file.
    • This tool checks the whole file's structure; it does not test a specific crawler against a specific URL path, which the site's separate robots.txt tester covers.

    What it takes

    • The full contents of a robots.txt file

    More tools

    All of them free, all of them browser-only.

    Browse the tools

    How to use it

    1. 1Paste the whole robots.txt fileInclude every line, not just the group you are focused on, since structural errors like a misplaced rule depend on what comes before it.
    2. 2Read the error and warning countAn error is a line that breaks the documented syntax outright; a warning is a line that parses but does something Google's documentation specifically discourages or does not support.
    3. 3Check any "no colon found" errors firstA missing colon is the most common copy-paste mistake, usually from an "Allow /path" line that dropped its colon.
    4. 4Check for rules before any user-agent lineAn allow or disallow line with no user-agent line above it has no group to belong to, per RFC 9309's group structure.
    5. 5Check sitemap lines are absolute URLsA relative sitemap path, such as sitemap.xml instead of a full https:// URL, fails Google's documented requirement for a fully qualified URL.
    6. 6Check for fields Google does not supportCrawl-delay and similar directives are real for some other crawlers but are flagged here because Google's documentation states it does not use them.

    Paste a full robots.txt file, and this tool checks whether it is well-formed at all: every line’s syntax, every field name, every path and URL value, and the group structure that ties rules to a user-agent. It runs entirely in your browser, line by line, against RFC 9309 and Google’s own documented interpretation of it.

    A different question from testing one crawler against one path

    The site’s other robots.txt tool answers “does this file allow Googlebot to crawl this one URL,” tracing through group selection and the longest-match rule for a specific case. This tool answers a question that comes before that one: is the file itself syntactically sound, independent of any particular crawler or path. A file can have a structural error, such as a disallow rule with no user-agent line above it, that never shows up when testing a single path against a single crawler, because the broken line simply gets ignored rather than throwing a visible error. Linting the whole file catches that class of mistake directly, rather than hoping a spot check happens to land on the broken line.

    What “valid” means here, precisely

    RFC 9309 defines a robots.txt line as a field, a colon, and a value, grouped under whichever user-agent lines precede it. Google’s own documentation adds the specifics this tool checks against: field names are case-insensitive but their values are usually case-sensitive, a path value “must start with / to designate the root,” and a sitemap value “must be a fully qualified URL, including the protocol and host.” The documentation is also explicit about what it ignores outright: “Google ignores invalid lines in robots.txt files… and use only valid lines,” which means a broken line rarely blocks the rest of the file, it just silently does nothing. That silence is exactly why a dedicated syntax check is worth running before trusting a robots.txt file in production, rather than assuming a file with no visible crawl errors must be well-formed.

    Sources

    1. 1.RFC 9309: Robots Exclusion Protocol - IETFPrimary
    2. 2.How Google interprets the robots.txt specification - Google Search Central
    Tools

    Hreflang XML sitemap generator

    Build a sitemap with xhtml:link hreflang annotations for localized pages, with every alternate listed on every URL as Google's documentation requires.