Skip to content
SEO Madmanby Adam Hafez
Free tool

robots.txt and llms.txt generator with an AI policy

Decide which AI crawlers may take your content, see what each one actually does, and generate both files from that one decision.

Runs entirely in your browser. Nothing you enter is uploaded, logged or stored.

Who may take your content

Blocking a training crawler does not remove you from the answer engine that shares its name, and allowing a search crawler does not consent to training. They are separate agents, which is why they are listed separately.

robots.txt

llms.txt
Key takeaways
  • A single company usually runs several crawlers with different purposes, and refusing one of them says nothing about the others.
  • Blocking a training crawler does not remove you from the answer engine that shares its name, because that engine reads with a different agent.
  • Google-Extended controls Gemini training only, and refusing it has no effect on how the page ranks in Google Search.
  • robots.txt states intent, so a bot rule at your CDN can silently overrule everything the file says.
What it takes
  • Your site URL
  • Paths you want kept out of results
  • A yes or no per crawler
More tools
All of them free, all of them browser-only.
Browse the tools
How to use it
  1. 1Set the site URL and the paths to excludeSearch pages and anything behind a query string are the usual candidates, because they generate near-infinite low-value URLs.
  2. 2Choose a starting postureThree presets: allow everything, allow search but refuse training, or refuse the AI crawlers entirely. Then adjust individual agents.
  3. 3Read the purpose label on each agentSearch, training and live fetch are genuinely different exchanges. Deciding them as one group is how sites end up invisible in answers they wanted to appear in.
  4. 4Publish both files at the rootrobots.txt goes at /robots.txt, llms.txt at /llms.txt, and then verify the policy with a real request rather than assuming.
Three kinds of crawler, three different exchanges


A search agent indexes your page so it can be cited in an answer. You get attribution and,
sometimes, a click. This is the closest thing to the traditional search bargain.


A training agent copies your content into a model. There is no attribution and no click. The
argument for allowing it is influence over what models know; the argument against is that you get
nothing back that you can measure.


A live fetcher retrieves your page because a person asked about it right now. It behaves like
a visitor with a robot’s user agent, and blocking it mostly means the person gets a worse answer
about you.


Deciding all three with one checkbox is the most common mistake in AI crawler policy, and it is
usually made in the direction of blocking search agents by accident.


robots.txt is a request, not a wall


Nothing in the protocol enforces anything. Well-behaved crawlers read the file and comply; the
rest do not. More importantly for most sites, your own infrastructure can overrule it: a bot
management rule, an AI crawler toggle at the CDN, or a WAF rule matching on bot signatures will
block an agent that robots.txt explicitly welcomes.


So verify with a request rather than a file:


for ua in GPTBot OAI-SearchBot ClaudeBot PerplexityBot; do
  curl -sA "$ua" -o /dev/null -w "$ua %{http_code}\n" https://example.com/
done


A 403 there means the decision is being made somewhere other than the file you just published.
Sources
  1. 1.RFC 9309: Robots Exclusion Protocol - IETFPrimary
  2. 2.Overview of Google crawlers and fetchers - Google Search CentralPrimary
  3. 3.OpenAI bots and how to control them - OpenAIPrimary
  4. 4.The llms.txt proposal - llmstxt.org
Frequently asked questions
If I block GPTBot, do I disappear from ChatGPT?
No. GPTBot collects training data. ChatGPT search results are built by OAI-SearchBot, and a page fetched because a user asked about it comes through ChatGPT-User. Blocking the training crawler leaves the other two untouched, which is usually what a publisher actually wants.
Does refusing Google-Extended hurt my rankings?
No. Google-Extended governs whether your content is used to train Gemini. It is a separate control from Googlebot, and Google documents that it does not affect Search ranking.
Is llms.txt a standard that engines obey?
It is a proposal, not a standard, and no engine is obliged to read it. It costs almost nothing to publish and it is the only place you get to state, in your own words, what a machine should read first. Treat it as a courtesy that may pay off, not as a ranking mechanism.
My robots.txt allows these bots but they still get blocked. Why?
Almost always a bot-management rule at the CDN. Cloudflare's AI crawler controls, bot fight modes and WAF rules that match on bot signatures all act before robots.txt is ever read. Test with a real request using the agent's user-agent string.