What is llms.txt?
llms.txt is a markdown file at the root of your site that describes what the site is and links to the pages worth reading. It is a proposed convention, not a standard, and no major model provider guarantees it is read. It costs an hour to write and it makes your positioning unambiguous in one fetchable place, which is why it is worth doing anyway.
What it is not
It is not robots.txt for AI. It does not grant or deny access — robots.txt still does that. It does not guarantee inclusion in any model's answers, and there is no evidence it changes ranking or citation probability on its own.
Anyone selling llms.txt as an AI visibility fix is overselling it. Treat it as documentation for machines, not as a growth lever.
Why write one anyway
Because the alternative is that a model infers your positioning from whatever page it happens to fetch. A single markdown file stating what you are, what you are not, and which pages matter removes that ambiguity cheaply — and the exercise of writing it usually exposes inconsistencies in how you describe yourself elsewhere.
It is also genuinely useful for the growing number of agents that fetch a site before acting on it.
The format
Keep it flat markdown. An H1 with your site name, a blockquote one-line summary, a few paragraphs of plain description, then H2 sections each containing a link list in the form of a link followed by a colon and a short description.
Include public marketing, documentation and content pages. Exclude login, account, admin, API and webhook routes — anything behind a session has no business in a file designed to be fetched by strangers.
Common mistakes
Listing every URL on the site, which buries the pages that matter. Writing marketing copy instead of description, which reads as noise. Letting it go stale after a redesign, so it points at pages that no longer exist. And listing authenticated routes, which is the one mistake with an actual security cost.
What we measured on real domains
Everything below comes from ThoughtForge runs against live domains, not from a study we read. Each run scores three things out of 100: identity (can an engine say who you are without contradicting itself), durability (does your brand survive a multi-turn conversation), and retrieval (can your pages be chunked into something quotable).
Across every domain we have scanned, one number is consistently the worst, and it is not the one people optimise for. Durability sits at or near zero almost everywhere. Brands that get named in turn one disappear by turn three, because nothing on the site gives the model a reason to keep holding onto them.
- thoughtforge.app, July: identity 16, durability 16, retrieval 49, overall 27.
- thoughtforge.app, September, after entity and schema work: identity 52, durability 0, retrieval 48, overall 33.
- A services domain (megap•••••••••••.com.au): identity 28, durability 0, retrieval 47, overall 25.
- An industry association (propertyasso•••••••.com.au): identity 4, durability 0, retrieval 50, overall 18.
- A consultancy (obse•••••••••.com): identity 40 in July, 52 in September; overall 30 to 35.
Does llms.txt change your score?
Honestly: not on its own. We publish one, and adding it moved none of the three scores on our own domain. What moved identity from 16 to 52 was schema consistency and crawler access, not the file.
Treat llms.txt as cheap insurance and a statement of intent rather than a lever. It costs twenty minutes, it may matter more later, and it will not rescue a site whose identity score is in single digits.
What to put in it if you publish one
Keep it short and honest: what the organisation is in one sentence, links to the handful of pages you would actually want quoted, and any usage terms you want stated. A file listing every URL on the site is a sitemap with extra steps.
Point it at pages that are already answer-first. Directing an engine to a page it cannot quote wastes the only attention you were given.
Adoption, honestly
No major engine has committed to reading llms.txt as a ranking or retrieval input. It is a convention some tools respect and others ignore, and that may change in either direction.
Publish it because it costs twenty minutes and documents your intent, not because you expect it to move a number. Ours did not.
Frequently asked questions
Do ChatGPT or Claude read llms.txt?
There is no published confirmation that either uses it for retrieval or ranking. Some agents and crawlers do fetch it. Write one because it is cheap and clarifying, not because it is guaranteed to be read.
Where does the file go?
At the root, served at /llms.txt, alongside robots.txt.
Does it replace a sitemap?
No. A sitemap is for search engine crawling and is a real standard. llms.txt is a human-readable description with a curated link list.
Related guides
- AI crawlers and robots.txt — Every major AI crawler user-agent, what it feeds, and how to allow or block each one in robots.txt without losing visibility.
- Entity clarity and schema — How to make an AI model certain about what your brand is, using Organization and Person schema, sameAs links and consistent naming.
- AI search visibility — AI search visibility is how often answer engines mention and cite your brand. How to measure it properly, what moves it, and what to ignore.