AI crawlers and robots.txt

The user-agents that matter

The ones worth naming explicitly in robots.txt:

• GPTBot — OpenAI's crawler. Feeds both training and search indexing. Blocking it materially reduces your chance of being cited by ChatGPT.
• OAI-SearchBot — OpenAI's retrieval crawler for ChatGPT search specifically. Allow this one if you want to appear in answers.
• ChatGPT-User — fetches a page when a user or a tool explicitly asks for that URL. Blocking it breaks the case where someone pastes your link into ChatGPT.
• ClaudeBot and anthropic-ai — Anthropic's crawlers.
• PerplexityBot — Perplexity's crawler. Perplexity cites sources prominently, so this is often the highest-value one to allow.
• Google-Extended — controls whether your content trains Gemini and appears in some Google AI surfaces. It does not affect classic Google Search indexing.
• CCBot — Common Crawl, which many models train from indirectly.

A sensible default

For almost every business trying to be found, the right answer is to allow all of them on public pages and disallow the authenticated parts of the app. The traffic cost is negligible and the visibility cost of blocking is high.

The exception is a publisher whose content is the product. There, blocking training crawlers while allowing retrieval crawlers is a defensible position — you stay citable without donating your archive.

What robots.txt cannot do

It cannot stop a model that already trained on your content before you added the rule. It cannot stop a user pasting your text into a chat. And it is advisory — well-behaved crawlers honour it, others do not.

It also cannot help if your pages need JavaScript to render. A crawler that is allowed in and receives an empty shell is functionally blocked.

How to verify it is working

Fetch your own page with the crawler's user-agent string and read what comes back. If you see your headline and body text in the raw HTML, you are fine. If you see an empty div, you have a rendering problem, not a robots problem.

Then check your server logs for those user-agents. Real crawl activity is the only proof that the configuration is taking effect.

What we measured on real domains

Everything below comes from ThoughtForge runs against live domains, not from a study we read. Each run scores three things out of 100: identity (can an engine say who you are without contradicting itself), durability (does your brand survive a multi-turn conversation), and retrieval (can your pages be chunked into something quotable).

Across every domain we have scanned, one number is consistently the worst, and it is not the one people optimise for. Durability sits at or near zero almost everywhere. Brands that get named in turn one disappear by turn three, because nothing on the site gives the model a reason to keep holding onto them.

• thoughtforge.app, July: identity 16, durability 16, retrieval 49, overall 27.
• thoughtforge.app, September, after entity and schema work: identity 52, durability 0, retrieval 48, overall 33.
• A services domain (megap•••••••••••.com.au): identity 28, durability 0, retrieval 47, overall 25.
• An industry association (propertyasso•••••••.com.au): identity 4, durability 0, retrieval 50, overall 18.
• A consultancy (obse•••••••••.com): identity 40 in July, 52 in September; overall 30 to 35.

Crawler access is a gate, not a lever

Blocking AI crawlers guarantees a bad score; allowing them guarantees nothing. Every domain in the numbers above already allowed GPTBot, ClaudeBot and PerplexityBot, and they still scored between 18 and 35 overall.

The more common failure we see is subtler than a robots.txt block: the crawler is allowed, fetches the page, and receives an empty JavaScript shell. Our own homepage did exactly this — it served search engines 42 words and no structured data while every other page served full text. Check what the crawler receives, not just what robots.txt permits.

Verify what the crawler actually receives

Requesting your own pages with a crawler user agent takes a minute and regularly contradicts what people assume. A page that looks complete in a browser can return a near-empty document to a bot when the content is assembled by JavaScript.

Check three things on each important page: the response status, the word count of the returned HTML with tags stripped, and whether the structured data block is present. If the word count is under about 300, the crawler is not seeing your page, whatever robots.txt says.

Which bots to allow, and the trade-off

Allowing GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended is the default for anyone who wants to be cited. Blocking them protects content from training use but removes you from answers entirely.

There is no middle setting that gets you citations without access. Decide deliberately rather than inheriting whatever your platform shipped.

Frequently asked questions

Will blocking GPTBot protect my content?

Partially and going forward only. It prevents future crawling; it does nothing about content already in a trained model or content republished elsewhere.

Does Google-Extended affect Google Search rankings?

No. It controls Gemini training and some AI surfaces. Classic Search indexing is governed by Googlebot.

Should a small business block any AI crawler?

Usually not. For most small businesses the risk of being invisible in AI answers far outweighs the risk of being trained on.

Related guides

• What is llms.txt? — llms.txt is a plain-markdown file that tells AI assistants what your site is and where to read more. What it does, what it doesn't, and how to write one.
• Why AI doesn't mention your brand — Six concrete reasons ChatGPT and Perplexity skip your brand, how to tell which one applies to you, and the fix for each.
• AI search visibility — AI search visibility is how often answer engines mention and cite your brand. How to measure it properly, what moves it, and what to ignore.