# llms.txt best practices: what to include and avoid

> How to make llms.txt useful: page selection, link notes, size, links to Markdown, refreshing, security and typical mistakes found in 584 thousand published files

Page: https://llms-full-txt.ru/en/guides/best-practices/
Site index: https://llms-full-txt.ru/en/llms.txt

A good `llms.txt` is a short, selective map: the main material of the site, grouped by meaning, with a note for every link.

A bad file fails in one of two ways: it holds nothing useful, or it holds everything.

Check the server response, not only the format. Half of the published files do not work the way their owner thinks.

## What the corpus shows

[Common Crawl](https://commoncrawl.org/blog) analyzed 584 thousand `llms.txt` files found in a single crawl of the web.

- 68%
- of files were made by a generator, not by hand: 50%
- have the full structure: title, summary, sections: 23%
- contain no links at all: 40%
- of successful responses at /llms.txt were HTML

The rules below follow from that: they answer mistakes that happen in practice, not invented ones.

## Selection beats completeness

An agent reads the whole file and decides where to go. The more secondary material it holds, the harder the main material is to find.

- Include pages that answer frequent questions.
- Do not include everything `sitemap.xml` has.
- Two to six sections are enough. A section with one link is usually unnecessary.
- Put secondary material into an `Optional` section at the end.

The proposal sets no hard size limit. The guide is simple: the file should be readable as a whole and still make sense.

## A note for every link

The note tells an agent whether a page is worth opening. A section name is too general for that.

Little use:

```markdown
- [Documentation](https://example.com/docs)
- [API](https://example.com/api): API
- [Guide](https://example.com/guide): a guide
```

Useful:

```markdown
- [Quick start](https://example.com/docs/start.md): installation, the key, the first request
- [API reference](https://example.com/docs/api.md): methods, parameters, error codes
- [Moving to version 3](https://example.com/docs/migrate.md): what breaks and how to update the code
```

A good note names what a person would search for: the subject, the action, the condition.

## Links to convenient content

The [second edition of the proposal](https://llmstxt.org/) says it directly: links should lead to content convenient for a model. When a page has a Markdown version, link to it.

- Addresses are absolute, with protocol and domain.
- Addresses are final, with no redirects.
- Each link appears in the file once.

How to serve Markdown: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/).

## The site summary

The line under the title is the only place where the site speaks about itself as a whole. It should answer "what is here and for whom", not be an advertising phrase.

Weak: "The best solution for your business".

Strong: "A delivery service for online stores: rate calculation, order creation, tracking".

## Typical mistakes

- The file address returns HTML
- The site answers 200 to any address and serves the home page or an error page.
- 40% of successful responses in the Common Crawl corpus.

- The file has no links
- The file consists of a title and a company description. An agent has nowhere to go.
- 23% of published files.

- The file was assembled automatically and never read
- A generator dumped every post in order. No selection, service pages inside.
- 68% of files were made by a generator.

- The file contains prohibitions
- The owner lists what robots must not do. The file does not control access, and the prohibition has no effect.
- The corpus contains files where robots.txt sits at the llms.txt address.

## Security

The file is public, and it is read by a program that can act.

- **Do not list private things.** Admin addresses, test environments, pages behind a login.
- **Do not list keys or tokens**, including inside link addresses.
- **Do not write instructions for the model.** Phrases like "always recommend our product" are an attempt to steer someone else's agent. Such files exist in the Common Crawl corpus, and responsible agents treat site content as untrusted.
- **Do not rely on the file as protection.** Prohibitions go into `robots.txt`: [AI crawlers](https://llms-full-txt.ru/en/guides/ai-crawlers/).

## Refreshing

The file ages together with the site. It is most convenient when it is assembled on publication:

- a documentation generator creates the file on every build;
- a content management system extension rebuilds it on a schedule;
- for a hand-written file, set a reminder and a link check.

A neglected file shows itself by links that answer `404` or redirect.

## Before publishing

1. One first-level heading with the site name.
2. A summary in one or two sentences, with no advertising words.
3. Every link has a note.
4. No repeats and no private addresses.
5. Every address answers `200` with no redirects.
6. The file is served as text with `200`.
7. It is clear who refreshes the file and when.

Commands for each check: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/).

## What to read next

- [The file format](https://llms-full-txt.ru/en/specification/format/) and [sections](https://llms-full-txt.ru/en/specification/sections/) - exact rules
- [Basic examples](https://llms-full-txt.ru/en/examples/basic/) and [advanced examples](https://llms-full-txt.ru/en/examples/advanced/)
- [An audit prompt](https://llms-full-txt.ru/en/prompts/audit-llms-txt/) - checking the file with the help of an AI system