AI crawlers and robots.txt
There are two distinct kinds of AI crawler: training crawlers, which collect data to train future models, and retrieval crawlers, which fetch pages in real time to answer a user's question right now. Blocking the second kind removes you from AI answers entirely. Blocking the first kind does not.
The user-agents that matter
The ones worth naming explicitly in robots.txt:
- GPTBot — OpenAI's crawler. Feeds both training and search indexing. Blocking it materially reduces your chance of being cited by ChatGPT.
- OAI-SearchBot — OpenAI's retrieval crawler for ChatGPT search specifically. Allow this one if you want to appear in answers.
- ChatGPT-User — fetches a page when a user or a tool explicitly asks for that URL. Blocking it breaks the case where someone pastes your link into ChatGPT.
- ClaudeBot and anthropic-ai — Anthropic's crawlers.
- PerplexityBot — Perplexity's crawler. Perplexity cites sources prominently, so this is often the highest-value one to allow.
- Google-Extended — controls whether your content trains Gemini and appears in some Google AI surfaces. It does not affect classic Google Search indexing.
- CCBot — Common Crawl, which many models train from indirectly.
A sensible default
For almost every business trying to be found, the right answer is to allow all of them on public pages and disallow the authenticated parts of the app. The traffic cost is negligible and the visibility cost of blocking is high.
The exception is a publisher whose content is the product. There, blocking training crawlers while allowing retrieval crawlers is a defensible position — you stay citable without donating your archive.
What robots.txt cannot do
It cannot stop a model that already trained on your content before you added the rule. It cannot stop a user pasting your text into a chat. And it is advisory — well-behaved crawlers honour it, others do not.
It also cannot help if your pages need JavaScript to render. A crawler that is allowed in and receives an empty shell is functionally blocked.
How to verify it is working
Fetch your own page with the crawler's user-agent string and read what comes back. If you see your headline and body text in the raw HTML, you are fine. If you see an empty div, you have a rendering problem, not a robots problem.
Then check your server logs for those user-agents. Real crawl activity is the only proof that the configuration is taking effect.
Frequently asked questions
Will blocking GPTBot protect my content?
Partially and going forward only. It prevents future crawling; it does nothing about content already in a trained model or content republished elsewhere.
Does Google-Extended affect Google Search rankings?
No. It controls Gemini training and some AI surfaces. Classic Search indexing is governed by Googlebot.
Should a small business block any AI crawler?
Usually not. For most small businesses the risk of being invisible in AI answers far outweighs the risk of being trained on.
Related guides
- What is llms.txt? — llms.txt is a plain-markdown file that tells AI assistants what your site is and where to read more. What it does, what it doesn't, and how to write one.
- Why AI doesn't mention your brand — Six concrete reasons ChatGPT and Perplexity skip your brand, how to tell which one applies to you, and the fix for each.
- AI search visibility — AI search visibility is how often answer engines mention and cite your brand. How to measure it properly, what moves it, and what to ignore.