llms.txt best practices

Common Crawl analyzed 584 thousand llms.txt files found in a single crawl of the web.

of files were made by a generator, not by hand

68%

Common Crawl, Aug 31, 2026

have the full structure: title, summary, sections

50%

Common Crawl, Aug 31, 2026

contain no links at all

23%

Common Crawl, Aug 31, 2026

of successful responses at /llms.txt were HTML

40%

Common Crawl, Aug 31, 2026

The rules below follow from that: they answer mistakes that happen in practice, not invented ones.

An agent reads the whole file and decides where to go. The more secondary material it holds, the harder the main material is to find.

• Include pages that answer frequent questions.
• Do not include everything sitemap.xml has.
• Two to six sections are enough. A section with one link is usually unnecessary.
• Put secondary material into an Optional section at the end.

The proposal sets no hard size limit. The guide is simple: the file should be readable as a whole and still make sense.

The note tells an agent whether a page is worth opening. A section name is too general for that.

Little use:

- [Documentation](https://example.com/docs)
- [API](https://example.com/api): API
- [Guide](https://example.com/guide): a guide

Useful:

- [Quick start](https://example.com/docs/start.md): installation, the key, the first request
- [API reference](https://example.com/docs/api.md): methods, parameters, error codes
- [Moving to version 3](https://example.com/docs/migrate.md): what breaks and how to update the code

A good note names what a person would search for: the subject, the action, the condition.

The second edition of the proposal says it directly: links should lead to content convenient for a model. When a page has a Markdown version, link to it.

• Addresses are absolute, with protocol and domain.
• Addresses are final, with no redirects.
• Each link appears in the file once.

How to serve Markdown: Markdown versions of pages.

The line under the title is the only place where the site speaks about itself as a whole. It should answer “what is here and for whom”, not be an advertising phrase.

Weak: “The best solution for your business”.

Strong: “A delivery service for online stores: rate calculation, order creation, tracking”.

When to use

The file address returns HTML

Trap

The site answers 200 to any address and serves the home page or an error page.

Seen in the wild

40% of successful responses in the Common Crawl corpus.

When to use

The file has no links

Trap

The file consists of a title and a company description. An agent has nowhere to go.

Seen in the wild

23% of published files.

When to use

The file was assembled automatically and never read

Trap

A generator dumped every post in order. No selection, service pages inside.

Seen in the wild

68% of files were made by a generator.

When to use

The file contains prohibitions

Trap

The owner lists what robots must not do. The file does not control access, and the prohibition has no effect.

Seen in the wild

The corpus contains files where robots.txt sits at the llms.txt address.

The file is public, and it is read by a program that can act.

• Do not list private things. Admin addresses, test environments, pages behind a login.
• Do not list keys or tokens, including inside link addresses.
• Do not write instructions for the model. Phrases like “always recommend our product” are an attempt to steer someone else’s agent. Such files exist in the Common Crawl corpus, and responsible agents treat site content as untrusted.
• Do not rely on the file as protection. Prohibitions go into robots.txt: AI crawlers.

The file ages together with the site. It is most convenient when it is assembled on publication:

• a documentation generator creates the file on every build;
• a content management system extension rebuilds it on a schedule;
• for a hand-written file, set a reminder and a link check.

A neglected file shows itself by links that answer 404 or redirect.

Commands for each check: Validating files and markup.

• The file format and sections - exact rules
• Basic examples and advanced examples
• An audit prompt - checking the file with the help of an AI system