What llms.txt is
A Markdown file at /llms.txt that tells language models what a site is and which pages to read, ideally as clean Markdown. It was proposed in September 2024 at llmstxt.org. It is a proposal, not a standard: no search engine needs it to index your site, and it does not control crawling. Blocking or allowing AI crawlers is still done in robots.txt.
The format
| Part | Required | Example |
|---|---|---|
| H1 with the site or project name | Yes, first line | # Example Docs |
| Blockquote summary | No | > Docs for the Example API. |
| Details: paragraphs or lists, no headings | No | Read the quickstart first. |
| H2 sections with link lists | No | ## Docs- [Quickstart](https://example.com/quickstart.md): first call |
## Optional section | No | Links an agent can skip when it needs a short context. |
The most common fault
Many sites answer /llms.txt with their normal HTML page or a "not found" page and HTTP 200, usually from a single-page app or CMS catch-all route. The file looks present but contains no Markdown. This checker flags that.
What this checker does not do
- It does not fetch the linked pages. Test any of them with the noindex checker.
- It does not look for
/llms-full.txt, a related convention for the full text in one file. - It cannot tell you whether a given AI product reads the file. To see which AI crawlers your robots.txt lets in, use the AI bot checker.
We fetch the file once, as NoindexWatchBot. Results are not stored.