Skip to content
SEO Madmanby Adam Hafez
Free tool

robots.txt tester: check a user-agent against a path

Paste a robots.txt, pick a crawler and a URL path, and see the exact Allow or Disallow line that decides the outcome.

Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.

Nothing you paste here is sent anywhere; it is parsed in your browser only.

    Key takeaways

    • The most specific user-agent group applies, and the tool only falls back to the wildcard group when no named group matches.
    • Within a matched group, the longest matching Allow or Disallow path wins, not whichever rule happens to appear first or last in the file.
    • An Allow and a Disallow of equal length resolve in favor of the Allow rule, per RFC 9309.
    • This tester checks an existing robots.txt from the testing direction, which complements the site's generator tool that writes a new file from a crawler policy.
    • Wildcard characters match any sequence of characters, and a trailing dollar sign anchors a rule to the exact end of the URL.

    What it takes

    • Existing robots.txt content
    • User-agent to test
    • URL path to test

    More tools

    All of them free, all of them browser-only.

    Browse the tools

    How to use it

    1. 1Paste the robots.txt you want to checkUse the file already live on your site, or a draft you are about to publish. The example prefilled here shows a wildcard group and a named-crawler group so both matching paths are visible at once.
    2. 2Enter the crawler's user-agentUse the product token a crawler actually announces, such as Googlebot or GPTBot, not its full user-agent string.
    3. 3Enter the URL path to testA path starting with a slash, such as /private/page.html. Query strings matter too, since a rule can target them directly.
    4. 4Read the verdict and the trace beneath itThe tool names the group that applied and the exact line that decided the outcome, plus any shorter rule that also matched but lost.

    Publishing a robots.txt is easy. Knowing what it actually does to one crawler on one path is the part most people get wrong, because the rule that wins is not the rule they expect.

    Longest match wins, not first or last match

    Within the group that applies to a crawler, RFC 9309 says the rule with the most matching characters decides the outcome, regardless of where it sits in the file. A short Disallow: /admin followed later by a more specific Allow: /admin/public does not lose to whichever line comes last: the Allow rule is longer, so it wins, and a request for /admin/public/page.html is allowed even though /admin alone is blocked.

    This trips people up in both directions. A broad Disallow near the top of the file feels like it should win because it was written first; a narrow Allow added later feels like an afterthought. Neither position matters. Only length does, and when an Allow and a Disallow match with exactly the same number of characters, the specification breaks the tie in favor of the Allow.

    Group selection works the same way, one level up. A crawler that matches a named group, such as Googlebot, follows that group’s rules and ignores the wildcard group entirely, even if the wildcard group would have produced a different verdict. Only a crawler with no named group falls back to *.

    Testing, not generating

    The site’s other robots.txt tool asks a different question: given a policy decision about which AI crawlers may take your content, what should the file say? That tool writes a robots.txt and an llms.txt from a set of yes-or-no choices per crawler.

    This tool runs the file forward instead of writing it. Paste what already exists, or a draft before you publish it, and check what it actually permits for one crawler and one path. Use the generator to decide the policy; use this tool to confirm the file you ended up with encodes that policy correctly, especially once wildcards and multiple groups are involved and the interaction between rules stops being obvious on a read-through.

    Remember that robots.txt is a request a well-behaved crawler chooses to honor, not an access control mechanism. A verdict of disallowed here describes what the file says, not what every bot in the wild will do with it.

    Sources

    1. 1.RFC 9309: Robots Exclusion Protocol - IETFPrimary
    2. 2.How Google interprets the robots.txt specification - Google Search Central
    Tools

    Hreflang XML sitemap generator

    Build a sitemap with xhtml:link hreflang annotations for localized pages, with every alternate listed on every URL as Google's documentation requires.