Skip to content
SEO Madmanby Adam Hafez
Free tool

X-Robots-Tag header checker and validator

X-Robots-Tag header checker: validate rules, user-agent prefixes, max-snippet and preview values, dates and conflicts. Free, runs in your browser.

Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.

Checked in your browser against Google's documented rule list. Nothing you type is sent anywhere.

Key takeaways

  • Google documents X-Robots-Tag as accepting the same rules as the robots meta tag, with an optional user-agent name before them, as in X-Robots-Tag: googlebot: nofollow.
  • Google states that the header name, user agent name and rule values are case-insensitive, and that several headers or comma-separated rules both work.
  • When robots rules conflict, Google says the more restrictive rule applies, so nosnippet overrides a max-snippet value on the same URL.
  • Google states that if robots.txt disallows a URL, the crawler never fetches it, so an X-Robots-Tag on that response is ignored.
  • This validator checks header text typed in and cannot fetch a live response, so it cannot see what your server actually sends.

What it takes

  • One or more pasted X-Robots-Tag header lines

More tools

All of them free, all of them browser-only.

Browse the tools

How to use it

  1. 1Copy the header linesTake each X-Robots-Tag line from your response headers, for example from browser developer tools or a curl request.
  2. 2Check the targetConfirm whether a user-agent prefix such as googlebot is present, since a prefixed line applies only to that crawler.
  3. 3Fix undocumented rulesReplace any rule flagged as not in Google's list, and correct numeric or enum values such as max-image-preview.
  4. 4Resolve conflictsWhere a looser rule sits beside a stricter one, remove the looser rule because the stricter one wins anyway.
  5. 5Confirm robots.txt allows crawlingThe header is only read if the URL is crawlable, so a disallow rule in robots.txt makes the header ineffective.

X-Robots-Tag carries robots rules in the HTTP response instead of the page’s HTML. Paste the header lines your server sends and this checker validates each rule against Google’s specification, in your browser.

What this X-Robots-Tag header checker checks

Each line must start with X-Robots-Tag:. The name is matched without regard to case, as Google states the header name, the user agent name and the values are not case-sensitive. The checker then reads:

  • The target. Google allows a user-agent name before the rules, as in X-Robots-Tag: googlebot: nofollow. With no name, the line applies to all crawlers.
  • Each rule. Rules are split on commas, which Google allows alongside separate header lines. Each is compared with Google’s list: all, noindex, nofollow, none, nosnippet, indexifembedded, max-snippet, max-image-preview, max-video-preview, notranslate, noimageindex and unavailable_after.
  • Values. max-snippet and max-video-preview take a whole number, 0 or more, or -1. Google says max-snippet:0 shows no snippet, like nosnippet, and -1 lets Google choose the length. max-image-preview takes none, standard or large. unavailable_after takes a date, and Google accepts RFC 822, RFC 850 and ISO 8601 formats. Flag rules such as noindex take no value.
  • Conflicts within a line. nosnippet beside max-snippet, or all beside noindex, nofollow or none. Google says the more restrictive rule applies.
  • Redundancy. none beside noindex or nofollow, since none already means both.

A final row reminds you that Google ignores the header on any URL robots.txt blocks from crawling.

How to read the results

  • Problem. The line is not an X-Robots-Tag header, it has no rules, a rule is not on Google’s list, a value is out of range, or a date does not parse.
  • Warning. A flag rule was given a value, rules conflict, or the unavailable_after date has already passed.
  • Info. The target of each line, redundant rules, and the robots.txt reminder.
  • Pass. A documented rule with a valid value, or an unavailable_after date in the future.

Worked example

The tool loads with three lines:

X-Robots-Tag: noindex, max-snippet:50, nosnippet
X-Robots-Tag: googlebot: max-image-preview:huge
X-Robots-Tag: unavailable_after: 2026-12-31T23:59:00+00:00

Line 1 targets all crawlers. noindex, max-snippet at 50 and nosnippet each pass on their own, then a conflict warning follows: nosnippet wins, so max-snippet:50 does nothing. Line 2 targets googlebot, and huge is a Problem because only none, standard and large are documented. Line 3 targets all crawlers. unavailable_after is not mistaken for a user-agent name, and the date passes as it is still in the future. The robots.txt reminder closes the list.

To fix it, remove max-snippet:50 from line 1 and change huge to large on line 2.

Common mistakes it catches

  • Rules that are not on Google’s list. Anything outside the twelve documented rules is flagged, including rules you may have copied from older examples.
  • Invented enum values. max-image-preview:huge or max-image-preview:full.
  • Contradictory rules. A snippet length set next to nosnippet has no effect.
  • Expired unavailable_after dates. A past date means the page is already due to drop from results.
  • A header on a blocked URL. If robots.txt disallows the URL, Google never sees the header.

The checker compares rules within one line only. If one header says nosnippet and another says max-snippet:50, read them together yourself.

Sources

  1. 1.Robots meta tag, data-nosnippet, and X-Robots-Tag specifications - Google Search CentralPrimary

Frequently asked questions

When should I use X-Robots-Tag instead of a meta tag?

The header works for files that have no HTML head, such as PDFs and images, because it is sent with the response rather than written in the page.

Does none mean the same as noindex, nofollow?

Yes. Google documents none as equivalent to noindex, nofollow, so listing both alongside none is redundant.

Why is max-image-preview:huge flagged?

Google documents only none, standard and large as max-image-preview values, so huge is not part of the documented syntax.

Why is noarchive flagged as not documented?

The checker uses the rule list on Google's current robots meta tag page, which names all, noindex, nofollow, none, nosnippet, indexifembedded, max-snippet, max-image-preview, max-video-preview, notranslate, noimageindex and unavailable_after. A rule outside that list is reported as a Problem.

How do I see the X-Robots-Tag my server sends?

Open the URL with your browser's developer tools and read the response headers in the Network panel, or request it with a command line tool such as curl. Copy each X-Robots-Tag line into the checker.

Tools

Hreflang XML sitemap generator

Hreflang sitemap generator: turn language codes and URLs into sitemap XML with xhtml:link alternates, and check codes and URLs. Free, runs in your browser.