Skip to content
SEO Madmanby Adam Hafez
Free tool

AI crawler robots.txt checker for GPTBot, ClaudeBot and more

Paste a robots.txt to see which AI crawlers it blocks: GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more. Free, runs in your browser.

Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.

Nothing you paste here is sent anywhere; it is parsed in your browser only.

    What the vendors say about six of these tokens

    • ChatGPT-User: OpenAI says requests it starts for a user may not follow robots.txt rules.
    • Perplexity-User: Perplexity says this fetcher generally ignores robots.txt rules, because a user requested the page.
    • Google-Extended: Google says it has no separate user agent string, controls Gemini training and grounding, and does not affect Search.
    • Applebot: Apple says Applebot follows the Googlebot group when no group names Applebot.
    • Applebot-Extended: Apple says it does not crawl pages; it only controls how data crawled by Applebot is used.
    • CCBot: Common Crawl says some crawlers falsely identify as CCBot, so verify user agent strings.

    Key takeaways

    • The checker covers 12 tokens, each one taken from the crawler documentation that OpenAI, Anthropic, Google, Perplexity, Apple and Common Crawl publish.
    • A crawler follows only the group that names its token, so a GPTBot group means GPTBot ignores every rule under the * group.
    • Tokens match exactly, so a group for Applebot does not apply to Applebot-Extended, and a group for GPTBot does nothing for ChatGPT-User.
    • Within a group the longest matching rule wins, and an Allow beats a Disallow of the same length, per RFC 9309.
    • OpenAI and Perplexity say their user-triggered fetchers, ChatGPT-User and Perplexity-User, may not follow robots.txt rules.

    What it takes

    • A pasted robots.txt file, edited directly in the textarea
    • A URL path or a full URL to check, with / as the default

    More tools

    All of them free, all of them browser-only.

    Browse the tools

    How to use it

    1. 1Paste your robots.txtCopy the contents of yourdomain.com/robots.txt into the box. This tool does not fetch URLs, so nothing you paste is sent anywhere.
    2. 2Enter a path to checkUse a path that starts with a slash, such as /blog/ai-seo, or a full URL. The default / checks the root of the site.
    3. 3Read the whole site columnBlocked means the root path is disallowed. Partly blocked means a Disallow rule applies to some paths. Allowed means no Disallow rule applies.
    4. 4Check which group each crawler usedThe line under each token says whether it used its own group, fell back to the * group, or followed the Googlebot group, which only Applebot does.
    5. 5Fix any flagged group nameA warning means a group names a token that is not documented here. Compare the spelling with the vendor page before you rely on that group.

    Paste a robots.txt and this checker reads it the way a crawler does, then tells you what each documented AI crawler is allowed to fetch. A misspelled token, or a group that overrides the * rules, shows up here before you assume a bot is blocked.

    What this checker tests

    It checks the 12 tokens in the table below, each one documented on its vendor’s own page. For every token it reports three things:

    • Whole site. Blocked when the root path / is disallowed, Partly blocked when a Disallow rule applies to some paths but not the root, and Allowed when no Disallow rule applies.
    • Your path. Allowed or Blocked for the path or URL you enter, with the rule that decided it.
    • Group used. Whether the crawler matched its own group, fell back to the * group, followed the Googlebot group, or matched no group at all.

    A summary counts how many of the 12 are blocked, and a warning lists any group that names a token which is not documented here.

    The crawlers it checks

    Purpose is worded as each vendor describes it. The tool does not claim a crawler obeys robots.txt beyond what its vendor says.

    TokenOperatorPurposeVendor page
    GPTBotOpenAITrainingOpenAI crawlers
    OAI-SearchBotOpenAISearchOpenAI crawlers
    ChatGPT-UserOpenAIUser-triggeredOpenAI crawlers
    ClaudeBotAnthropicTrainingAnthropic crawler policy
    Claude-SearchBotAnthropicSearchAnthropic crawler policy
    Claude-UserAnthropicUser-triggeredAnthropic crawler policy
    Google-ExtendedGoogleGemini training and groundingGoogle common crawlers
    PerplexityBotPerplexitySearchPerplexity crawlers
    Perplexity-UserPerplexityUser-triggeredPerplexity crawlers
    ApplebotAppleSearchAbout Applebot
    Applebot-ExtendedAppleTraining controlAbout Applebot
    CCBotCommon CrawlOpen web crawl datasetCCBot

    How the matching works

    The tool applies the rules in RFC 9309, the Robots Exclusion Protocol standard:

    • One group per crawler. A crawler uses the group or groups that name its token, compared case-insensitively. Only if none does, it uses the * group. Groups that name the same token are merged.
    • Exact tokens. Applebot and Applebot-Extended are different tokens, so one group does not cover the other.
    • Longest rule wins. The matching Allow or Disallow with the longest path decides, wherever it sits in the file. An Allow beats a Disallow of the same length.
    • Wildcards. A * in a path matches any run of characters, and a $ at the end anchors the rule to the end of the URL.
    • Defaults. No matching rule means allowed, and an empty Disallow blocks nothing.
    • Applebot. Apple’s page says Applebot follows the Googlebot group when no group names Applebot, so the tool does the same for that one token.

    Worked example

    The pre-filled file has a * group, one group shared by GPTBot and Google-Extended, a ClaudeBot group with an Allow, and a group for ChatGPT:

    User-agent: *
Disallow: /admin/

User-agent: GPTBot
User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /
Allow: /blog/

User-agent: ChatGPT
Disallow: /

    With the path /, three crawlers are Blocked for the whole site: GPTBot, Google-Extended and ClaudeBot. The other nine fall back to the * group, so they are Partly blocked because Disallow: /admin/ applies to them. Change the path to /blog/ai-seo and ClaudeBot flips to Allowed, because Allow: /blog/ is six characters against one for Disallow: /. GPTBot and Google-Extended stay Blocked.

    The last group is flagged. ChatGPT is not a token OpenAI documents, which lists ChatGPT-User for user-triggered requests. That group changes nothing for any crawler above, so ChatGPT-User still falls back to *.

    Common mistakes it catches

    • Adding a bot group and forgetting the * rules. Once GPTBot has its own group, it ignores *, so a Disallow: /admin/ you wanted for everyone must be repeated in that group.
    • Blocking the training token and expecting search to go too. OpenAI says OAI-SearchBot and GPTBot are independent settings, and Anthropic lists ClaudeBot, Claude-SearchBot and Claude-User as three separate robots.
    • Misspelled or undocumented tokens. A group for a name no vendor documents is read like any other group, but it matches none of the crawlers this tool lists.
    • Expecting a group for one token to cover its sibling. Applebot does not cover Applebot-Extended.

    What it cannot tell you

    A Blocked result describes what your file says, not what every bot does. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. The tool also cannot see your server’s firewall or CDN rules, never fetches your live file, and checks only the 12 tokens above. For the syntax of the whole file, use the robots.txt syntax validator, and for one crawler and one path in detail, use the tester.

    Sources

    1. 1.RFC 9309: Robots Exclusion Protocol - IETFPrimary
    2. 2.Overview of OpenAI Crawlers - OpenAIPrimary
    3. 3.Does Anthropic crawl data from the web, and how can site owners block the crawler? - AnthropicPrimary
    4. 4.Google's common crawlers - Google Search CentralPrimary
    5. 5.Perplexity Crawlers - PerplexityPrimary
    6. 6.About Applebot - ApplePrimary
    7. 7.CCBot - Common CrawlPrimary

    Frequently asked questions

    Which AI crawlers does this checker cover?

    It covers 12 tokens from OpenAI, Anthropic, Google, Perplexity, Apple and Common Crawl. Each one is listed only after it was found on that vendor's own crawler page. Googlebot and Bingbot are search crawlers that are not AI-only, so they are not on the list.

    Does blocking GPTBot remove my site from ChatGPT search?

    No. OpenAI says each setting is independent. GPTBot is for training, and OAI-SearchBot is the token that controls whether a site appears in ChatGPT search results, so a site can allow one and disallow the other.

    Does Google-Extended affect my Google Search rankings?

    No. Google says Google-Extended controls whether crawled content may be used for Gemini training and grounding, and that it does not affect a site's inclusion in Google Search or act as a ranking signal.

    Will every crawler on the list obey my robots.txt?

    Not necessarily. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. RFC 9309 also says robots.txt is not a form of access authorization.

    Why does Applebot follow my Googlebot rules?

    Because Apple's documentation says so. If robots instructions do not mention Applebot but do mention Googlebot, Applebot follows the Googlebot instructions. The checker applies that fallback to Applebot only.

    Why does a group for Applebot not block Applebot-Extended?

    Tokens are matched exactly, ignoring case, as RFC 9309 describes for a crawler's product token. Applebot-Extended is a separate token that only controls how data crawled by Applebot is used, so it needs its own group.