# LLMs Full Text
> A reference on llms.txt: the format, v2 of the proposal, examples and tools
## llms.txt reference: format, examples and tools
URL: https://llms-full-txt.ru/en/
Search is changing: more and more people get the answer right inside ChatGPT, Gemini and other AI systems. This reference explains how to prepare a site for that world - from content structure and Markdown to llms.txt, crawlers and GEO.
[What to do with my site](https://llms-full-txt.ru/en/guides/ai-ready-site/)
Who reads a site now
- A person asks a question
- AI search or an agent looks for a source
- Your site serves content in a clear form HTML Markdown llms.txt structured data
- The answer with a link to the source
## Your site is now read not only by people
What has changed
Before
- A person types a query
- Search shows a list of links
- A site the person reads the page
Now
- A person asks a question
- AI search or an agent finds and reads pages by itself
- Sources several sites at once, yours among them
- A ready answer the person may not open a single page
A site used to be prepared for a browser and a search robot first. Now there is one more consumer: the AI agent. It also has to find the right page, understand its structure and get the content without noise.
## Where is llms.txt here?
A map of material for an AI agent
llms.txt is a file in the site root: the name, a short description and lists of links to what matters.
[File format](https://llms-full-txt.ru/en/specification/format/)
Three files, three different questions
- `robots.txt` where a robot may go
- `sitemap.xml` which pages exist
- `llms.txt` which material may be useful to an agent
This is a simplified model, not three interchangeable standards.
## One file is not enough to make a site ready for AI
A site ready for AI
Five parts of one task. llms.txt covers one of them.
- [Find](https://llms-full-txt.ru/en/guides/ai-crawlers/) Crawlers, robots.txt, links, indexability.
- [Understand](https://llms-full-txt.ru/en/guides/markdown-pages/) Clear page structure, headings, Markdown.
- [Choose](https://llms-full-txt.ru/en/specification/format/) llms.txt, docs navigation, files in subpaths.
- [Trust](https://llms-full-txt.ru/en/guides/does-llms-txt-work/) Sources, authorship, freshness, evidence.
- [Measure](https://llms-full-txt.ru/en/guides/ai-visibility/) Bot logs, AI referrals, mentions and citations.
## Does it help promotion at all?
What is known and what is not
Confirmed
- Large documentation sites publish llms.txt themselves.
- Agents do work with Markdown versions of pages.
- Lighthouse checks the file in the experimental Agentic Browsing category.
- AI bots visit sites: their owners describe it in their own documentation.
Not proven
- That having llms.txt improves search rankings.
- That the file guarantees a place in a ChatGPT answer.
- That it increases citations by itself.
llms.txt is one part of preparing a site for AI, not a magic SEO button.
[What is known about the benefit](https://llms-full-txt.ru/en/guides/does-llms-txt-work/)
## What “promotion in neural networks” means in practice
- GEO Generative Engine Optimization
- AEO Answer Engine Optimization
- AI SEO SEO for AI search
- AI visibility being visible in AI
One task
To be found, understood and used by an AI system
What AI visibility is made of
- Accessibility the page can be fetched
- Understanding the structure can be parsed
- Citable content there is something to quote
- Measurement you can see what happens
Many terms, one meaning. Nobody can promise a place in an AI answer.
[GEO and AEO without promises](https://llms-full-txt.ru/en/guides/geo-aeo/)
## A site can be prepared step by step
Check access
AI bots can fetch the pages of the site.
[AI crawlers](https://llms-full-txt.ru/en/guides/ai-crawlers/)
Tidy the content
Clear headings and page structure.
[Best practices](https://llms-full-txt.ru/en/guides/best-practices/)
Add Markdown
Where it is justified: documentation, help.
[Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/)
Create llms.txt
A correct file by the second edition.
[Getting started](https://llms-full-txt.ru/en/guides/getting-started/)
Check the file and links
Status code, structure, live addresses.
[Checklist](https://llms-full-txt.ru/en/guides/validation/)
Measure the result
Bot logs, AI referrals, mentions and citations.
[Measurement](https://llms-full-txt.ru/en/guides/ai-visibility/)
## You do not have to assemble everything by hand
For some CMS there are ready-made extensions: they collect the site structure, build llms.txt and keep it up to date.
How a ready-made solution works
- CMS pages, products, posts
- Extension reads what is published
- Site structure sections and links
- llms.txt built and kept fresh
- Check the same as for a hand-written file
- Webasyst
Site
A map of site pages
Data source
Published site pages, the site name and contacts
Creates
`llms.txt``llms-full.txt`
ready-made The base edition is free
[View the extension](https://www.webasyst.com/store/plugin/site/kaktak_llmstxt/?promo_id=BE462B3B6D)
- Webasyst
Shop-Script
A map of an online store
Data source
Categories, products, brands, information pages and contacts
Creates
`llms.txt``llms-full.txt`
ready-made
[View the extension](https://www.webasyst.com/store/plugin/shop/kaktak_llmstxt/?promo_id=BE462B3B6D)
- Webasyst
Blog
A map of blog posts
Data source
Published posts, categories and tags
Creates
`llms.txt``llms-full.txt`
ready-made
[View the extension](https://www.webasyst.com/store/plugin/blog/kaktak_llmstxt/?promo_id=BE462B3B6D)
- Webasyst
Hub
A map of community discussions
Data source
Public sections and published discussions
Creates
`llms.txt``llms-full.txt`
ready-made
[View the extension](https://www.webasyst.com/store/plugin/hub/kaktak_llmstxt/?promo_id=BE462B3B6D)
- Webasyst
Helpdesk
A map of a knowledge base
Data source
Public FAQ and pages; tickets and correspondence are left out
Creates
`llms.txt``llms-full.txt`
ready-made
[View the extension](https://www.webasyst.com/store/plugin/helpdesk/kaktak_llmstxt/?promo_id=BE462B3B6D)
- Webasyst
Photos
A map of photo albums
Data source
Public albums and pages
Creates
`llms.txt``llms-full.txt`
ready-made
[View the extension](https://www.webasyst.com/store/plugin/photos/kaktak_llmstxt/?promo_id=BE462B3B6D)
Promo code for a discount on your first Webasyst order: BE462B3B6D
Commercial extensions from the Webasyst Store. Each one builds llms.txt and llms-full.txt from the data of its app and rebuilds them on a schedule or on demand. A file made by an extension is checked the same way as one written by hand.
## Where to start for you
- Site owner
«I need to understand whether I should do anything at all»
- [A plain explanation](https://llms-full-txt.ru/en/guides/what-is-llms-txt/)
- [What is proven and what is not](https://llms-full-txt.ru/en/guides/does-llms-txt-work/)
- [How to check my site](https://llms-full-txt.ru/en/guides/validation/)
- SEO specialist and marketer
«I want to understand GEO, AI crawlers and measurement»
- [GEO and AEO](https://llms-full-txt.ru/en/guides/geo-aeo/)
- [AI crawlers and robots.txt](https://llms-full-txt.ru/en/guides/ai-crawlers/)
- [Status and evidence](https://llms-full-txt.ru/en/guides/does-llms-txt-work/)
- Developer
«I want to implement it correctly»
- [Format by the second edition](https://llms-full-txt.ru/en/specification/format/)
- [What changed in v2](https://llms-full-txt.ru/en/specification/changes/)
- [Advanced scenarios](https://llms-full-txt.ru/en/examples/advanced/)
## It is already used in real projects
These companies publish llms.txt or Markdown representations in their own documentation. It does not mean their agents read your file.
- [Anthropic docs. anthropic. com](https://docs.anthropic.com/llms.txt)
- [Cloudflare developers. cloudflare. com](https://developers.cloudflare.com/llms.txt)
- [Vercel vercel. com](https://vercel.com/llms.txt)
- [Yandex yandex. cloud](https://yandex.cloud/llms.txt)
- [Stripe docs. stripe. com](https://docs.stripe.com/llms.txt)
- [Supabase supabase. com](https://supabase.com/llms.txt)
llms.txt files found in an open web archive
584K
Common Crawl, August 2026
of sampled sites publish the file
2.1%
Web Almanac 2025
the second edition of the proposal
v2
llmstxt.org, August 10, 2026
[Real files explained](https://llms-full-txt.ru/en/examples/advanced/)
## See how it works
- [llms.txt in five minutes](https://llms-full-txt.ru/en/guides/getting-started/) A minimal file, where to put it and how to check it.
- Write the file
- Put it in the site root
- Check the server response
- [llms.txt and llms-full.txt](https://llms-full-txt.ru/en/guides/llms-full-txt/) An index and the full text: the difference.
`llms.txt` an index of links `llms-full.txt` all text in one file
- [AI crawlers](https://llms-full-txt.ru/en/guides/ai-crawlers/) Three kinds of bots, three different robots.txt decisions.
training search user request
- [llms.txt and MCP](https://llms-full-txt.ru/en/reference/mcp-relationship/) Two independent systems and a bridge.
a file a protocol
- [Real files](https://llms-full-txt.ru/en/examples/advanced/) How OpenAI, Stripe and others build theirs.
- [Markdown for agents](https://llms-full-txt.ru/en/guides/markdown-pages/) A clean version of a page at the same address.
- [Discovery in v2](https://llms-full-txt.ru/en/specification/discovery/) The alternate and describedby relations.
- [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/) Studies and their limits.
## What is changing right now
- September 18, 2026 event
Lighthouse 13.5.0 grouped the llms.txt check with other agent discovery points
For a site owner. The check stays in the report, now next to other ways an agent discovers a site.
[Read the note](https://llms-full-txt.ru/en/blog/lighthouse-13-5-agent-discovery/)
- August 31, 2026 study
Common Crawl analyzed 584 thousand files: 68% made by a generator, 23% without a single link
For a site owner. A file without links is useless: check the content, not just the presence.
[Read the note](https://llms-full-txt.ru/en/blog/common-crawl-584k-llms-txt/)
- August 10, 2026 edition
The second edition of the proposal: discovery relations and files in subpaths
For a site owner. The file can be declared in the page code, and sections can have files of their own.
[Read the note](https://llms-full-txt.ru/en/blog/llms-txt-v2/)
- June 15, 2026 study
Ahrefs: 97% of the files were never requested
For a site owner. The file does not bring bots by itself: read the server logs.
[Read the note](https://llms-full-txt.ru/en/blog/ahrefs-llms-txt-study/)
- June 15, 2026 event
Google documented it: Search does not use such files
For a site owner. Do not expect an effect on Google rankings.
[Read the note](https://llms-full-txt.ru/en/blog/google-search-ignores-llms-txt/)
- May 7, 2026 event
Lighthouse 13.3.0 started checking llms.txt by default
For a site owner. A missing file became visible in a regular page report.
[Read the note](https://llms-full-txt.ru/en/blog/lighthouse-checks-llms-txt/)
- October 3, 2026 [Ahrefs: 97% of llms.txt files were never requested](https://llms-full-txt.ru/en/blog/ahrefs-llms-txt-study/)
- October 3, 2026 [Chrome describes how to prepare a site for agents](https://llms-full-txt.ru/en/blog/chrome-agent-ready-toolkit/)
- October 3, 2026 [Cloudflare serves Markdown to agents instead of HTML](https://llms-full-txt.ru/en/blog/cloudflare-markdown-for-agents/)
- [All blog posts](https://llms-full-txt.ru/en/blog/)
## Need to prepare an existing site?
Audit and implementation are done by the kaktak.net team.
[Discuss the task](https://kaktak.net/)
[LLMs Full Text](https://llms-full-txt.ru/en/) A reference on llms.txt and AI-ready sites: format, examples, tools.
Start here
- [An AI-ready site](https://llms-full-txt.ru/en/guides/ai-ready-site/)
- [What is llms.txt](https://llms-full-txt.ru/en/guides/what-is-llms-txt/)
- [Getting Started](https://llms-full-txt.ru/en/guides/getting-started/)
- [Best Practices](https://llms-full-txt.ru/en/guides/best-practices/)
- [Status of llms.txt](https://llms-full-txt.ru/en/guides/standard-status/)
llms.txt
- [What is llms-full.txt](https://llms-full-txt.ru/en/guides/llms-full-txt/)
- [llms.txt file format](https://llms-full-txt.ru/en/specification/format/)
- [Sections and Structure](https://llms-full-txt.ru/en/specification/sections/)
- [Discovering the file and Markdown versions](https://llms-full-txt.ru/en/specification/discovery/)
- [Changes in v2](https://llms-full-txt.ru/en/specification/changes/)
- [Basic Examples](https://llms-full-txt.ru/en/examples/basic/)
- [Advanced Examples](https://llms-full-txt.ru/en/examples/advanced/)
A machine-readable site
- [Structured data and JSON-LD](https://llms-full-txt.ru/en/guides/json-ld/)
- [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/)
- [AI crawlers and robots.txt](https://llms-full-txt.ru/en/guides/ai-crawlers/)
- [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/)
- [Lighthouse and llms.txt](https://llms-full-txt.ru/en/guides/lighthouse/)
- [llms.txt in WordPress](https://llms-full-txt.ru/en/guides/wordpress/)
- [llms.txt and JSON-LD in Webasyst](https://llms-full-txt.ru/en/guides/webasyst/)
AI and search
- [AI Search Optimization](https://llms-full-txt.ru/en/guides/geo-aeo/)
- [How to measure AI visibility](https://llms-full-txt.ru/en/guides/ai-visibility/)
- [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/)
Tools
- [Generators and Tools](https://llms-full-txt.ru/en/tools/generators/)
- [llm.codes](https://llms-full-txt.ru/en/tools/llm-codes/)
- [llmstxt-architect](https://llms-full-txt.ru/en/tools/llmstxt-architect/)
- [llmstxt CLI](https://llms-full-txt.ru/en/tools/llmstxt-cli/)
Reference
- [Setup in documentation generators](https://llms-full-txt.ru/en/reference/configuration/)
- [llms.txt and MCP](https://llms-full-txt.ru/en/reference/mcp-relationship/)
- [Who publishes llms.txt](https://llms-full-txt.ru/en/reference/adoption/)
- [Glossary](https://llms-full-txt.ru/en/reference/glossary/)
- [Paying and working from Russia](https://llms-full-txt.ru/en/reference/from-russia/)
Prompts
- [All prompts](https://llms-full-txt.ru/en/prompts/)
- [Generate llms.txt](https://llms-full-txt.ru/en/prompts/generate-llms-txt/)
- [Audit llms.txt](https://llms-full-txt.ru/en/prompts/audit-llms-txt/)
- [Optimize content for AI](https://llms-full-txt.ru/en/prompts/optimize-for-ai/)
Project
- [Blog](https://llms-full-txt.ru/en/blog/)
- [llms.txt](https://llms-full-txt.ru/en/llms.txt)
- [llms-full.txt](https://llms-full-txt.ru/en/llms-full.txt)
- [RSS](https://llms-full-txt.ru/en/rss.xml)
© 2025-2026 LLMs Full Text
Built by
## An AI-ready site: llms.txt, markup and crawler access
URL: https://llms-full-txt.ru/en/guides/ai-ready-site/
Preparing a site for AI means a machine can fetch a page, learn whether it may read it, find the right address, understand what the page is about, and read the content itself.
One file does not do that. `llms.txt` is one layer out of five. Most sites already have the other four, in varying shape.
## Five layers, one question each
- What a machine learns at each layer
- access rules for robots
- the list of pages
- a map for an AI agent
- the entity, its properties and relations
- the content itself
The layers do not replace one another. `robots.txt` says nothing about what is useful. `llms.txt` forbids nothing. Structured data does not replace the text of a page. And the most detailed `llms.txt` will not help a site that answers with an error.
## Layer zero: the page has to arrive
Before any file comes a plain requirement: the server has to answer.
- The page returns `200`, not a redirect chain and not an error.
- The main text is in the HTML sent by the server. If content appears only after scripts run in a browser, some automated clients will never see it.
- The response is fast and stable. A fast, reliable response lowers friction for every client: a person, a search robot, an agent.
- Reading the text requires no clicks, no dismissed dialogs and no scrolling.
- A page has one permanent address, and links are ordinary links that can be followed.
Structured data and `llms.txt` do not fix a slow or unreachable site. They describe what can already be fetched.
## 1. robots.txt: may I go
`robots.txt` holds access rules. The site owner tells robots where they may go.
AI companies run robots of three different classes: some collect training data, some build a search index, some come on behalf of a specific user. Blocking training while staying in AI search takes different lines in the file.
What to check: the file returns `200`, the robots you need are not blocked by accident, and the decision for each class is deliberate.
More: [AI crawlers and robots.txt](https://llms-full-txt.ru/en/guides/ai-crawlers/).
## 2. sitemap.xml: which addresses exist
`sitemap.xml` lists the pages of a site and helps discover them. It is a complete list without judgment: the sitemap does not say what matters.
`sitemap.xml` and `llms.txt` do not compete. The first answers "what exists", the second "where to start".
What to check: the sitemap exists, is referenced from `robots.txt`, contains only working addresses and no pages closed to indexing.
## 3. llms.txt: where it is useful to go
`llms.txt` is a map of useful material for an AI agent: the site name, a short description and lists of links with notes.
The caveats matter:
- it is a proposal, not an adopted standard;
- search engines do not use it for ranking;
- the [second edition](https://llmstxt.org/) is current: it added discovery relations and files for separate sections of a site;
- `llms-full.txt` is not part of the proposal; it is an ecosystem convention.
What to check: the file sits in the site root, is served as text with `200`, and its links work.
More: [What is llms.txt](https://llms-full-txt.ru/en/guides/what-is-llms-txt/), [the file format](https://llms-full-txt.ru/en/specification/format/), [what is llms-full.txt](https://llms-full-txt.ru/en/guides/llms-full-txt/).
## 4. Structured data: what is on the page
`llms.txt` helps a machine decide where to go. Structured data states explicitly where it has arrived: this is a product, an article or a discussion, what it is called, who wrote it, what it costs, whether it is in stock.
The vocabulary for such a description is [Schema.org](https://schema.org/). The most convenient way to write it is JSON-LD: a block of data in the page code, separate from the visible text.
- Route, meaning and content
- llms.txt
- there is a useful page about a product here
- /products/widget/
- JSON-LD
- this is a Product: name, SKU, price, availability
- HTML / Markdown
- description, specifications, instructions
For search engines this is a working layer, not decoration. Google states that it uses structured data to understand the content of a page. Yandex reads product markup and, for product pages marked up as `Product` with an `Offer`, gives the data on the page the highest priority over other sources.
If Schema.org can describe the kind of page and the markup is missing, the machine-readable form of the page is unfinished.
More: [Structured data and JSON-LD](https://llms-full-txt.ru/en/guides/json-ld/).
## 5. HTML and Markdown: the content itself
Everything else leads here. A machine needs text: clear headings, a sensible structure, concrete facts, dates and authors.
Every site already has HTML. Some sites also serve a Markdown version of a page at the same address: no menus, no ads, no styling. That is easier for an agent because there is less noise in the text.
What to check: one first-level heading per page, sections named by meaning, key facts written as text rather than baked into an image.
More: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/), [best practices](https://llms-full-txt.ru/en/guides/best-practices/).
## Trust and measurement
Two questions run through every layer.
**Can this be trusted.** Sources, an author and an update date are stated. The price in the markup matches the price on the page. Claims can be verified.
**Does it work.** Four things are measurable: bot requests in server logs, referrals from AI systems in analytics, citations, and share of mentions over a fixed set of queries. A place in an AI answer cannot be measured in advance or guaranteed.
More: [How to measure AI visibility](https://llms-full-txt.ru/en/guides/ai-visibility/), [does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/).
## What is known and what is not
**Confirmed**
- Search engines read `robots.txt`, `sitemap.xml` and structured data: their documentation says so.
- Google and Yandex use product markup when showing price and availability.
- AI bots visit sites: their owners publish the bot names and purposes.
- Large documentation sites publish `llms.txt` and Markdown versions of pages.
**Not proven**
- That `llms.txt` affects search rankings or the number of citations.
- That a particular AI system reads the JSON-LD of other sites directly: it depends on how the system is built, and only its own documentation can settle it.
**What it does not mean**
- Markup is not required for indexing: a page will be found without it.
- The value of structured data for AI is not limited to direct reading. AI search and agents often get facts through search engines, and those do read the markup.
## A step-by-step check
1. Pages return `200`, and the main text is in the server-sent HTML.
2. `robots.txt` exists, and the decision about AI bots is deliberate.
3. `sitemap.xml` exists and is referenced from `robots.txt`.
4. Pages carry structured data of a fitting type, and the Google and Yandex validators report no errors.
5. Pages have clear headings and structure.
6. `llms.txt` sits in the site root, and its links work.
7. Price, availability and dates in the markup match what a person sees.
8. Server logs show which bots come.
How to check each item: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/).
## What can be automated
Some layers are assembled from data the content management system already holds: the list of pages, products, posts, authors. It makes sense to hand that work to an extension that rebuilds the files and markup on every change.
Other things cannot be automated: clear writing, accurate facts, and the decision about which robots to admit.
How it is built in one system: [llms.txt and JSON-LD in Webasyst](https://llms-full-txt.ru/en/guides/webasyst/). All the extensions are listed on the developer page in the Webasyst Store. For WordPress: [llms.txt in WordPress](https://llms-full-txt.ru/en/guides/wordpress/).
## GEO and AEO: optimizing a site for AI search
URL: https://llms-full-txt.ru/en/guides/geo-aeo/
GEO and AEO are names for one task: getting a site into the answers of AI systems and AI search, and getting it cited.
GEO came from a 2023 research paper. AEO and the other words were coined by service vendors. Google treats all of it as ordinary SEO.
No single technique guarantees a place in an answer. What works is what worked before: an accessible site, clear text, verifiable facts.
## Where the words came from
- Four names for one task
- A research paper
- GEO
- Generative Engine Optimization: an arXiv paper, November 2023, accepted to KDD 2024
- Vendor words
- AEO
- Answer Engine Optimization: attributed to 2017, the primary source is not confirmed
- LLMO
- a synonym of GEO, no primary source found
- AI SEO
- two meanings: AI for SEO, and visibility in AI
- There is no difference of substance between them.
In the original GEO paper the authors changed the text of pages and measured how often a source appeared in the answer of a generative system. The best techniques raised visibility by up to 40% on their query set. That is an experiment under laboratory conditions, not a promise for any site.
Definitions of all the terms: [Glossary](https://llms-full-txt.ru/en/reference/glossary/).
## What Google says
In its guide for site owners [Google writes](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) two things:
- AEO and GEO are terms you may see used, while optimizing for AI features of Search is still SEO;
- there is no need to create special machine-readable files for Search: Search does not use them.
This is one search engine's position about its own products. It says nothing about how other AI systems work.
## How an answer is assembled
- From a question to an answer with a link
- A question
- a person asks an AI system
- Search
- the system looks for pages in a search index
- Reading
- an agent opens several pages
- An answer
- text with links to sources
The main point follows from the scheme: to get into an answer, a page first has to be found. Most AI systems with search rely on an ordinary search index. A site missing from search will not appear in an answer either.
## What affects it
The fundamentals are known reliably. They are the same as for search.
- **The page is reachable.** It answers `200`, the text is in the server-sent HTML, the robots you want are not blocked.
- **The page is indexed.** A search engine knows it.
- **The page has a direct answer.** A definition, a number, a step - as text, not as a hint.
- **The text is clearly built.** Headings by meaning, lists and tables where they fit.
- **Facts are verifiable.** The source, the author and the date are stated.
- **Entities are described explicitly.** Structured data says that this is a product, an article or an organization.
There is no need to write "for machines" at the expense of people. Clear text is read by both.
## Where files and markup fit
- **`robots.txt`** decides whether a robot is let in. Blocking training while staying in AI search is possible: those are different robots. [AI crawlers](https://llms-full-txt.ru/en/guides/ai-crawlers/).
- **Structured data** is read by search engines, and through them the facts reach AI systems. [Structured data and JSON-LD](https://llms-full-txt.ru/en/guides/json-ld/).
- **`llms.txt`** helps an agent that has already arrived find what it needs. According to the available studies, it does not affect appearing in answers. [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/).
- **Markdown versions** make reading easier for an agent. [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/).
Every layer in order: [An AI-ready site](https://llms-full-txt.ru/en/guides/ai-ready-site/).
## What is known and what is not
**Confirmed**
- Google applies the same fundamentals to AI features of Search as to ordinary search.
- The GEO research paper showed that text edits change how often a source appears in an answer under experimental conditions.
- AI systems publish the names and purposes of their robots.
**Not proven**
- That having `llms.txt` increases citations: the [SE Ranking](https://seranking.com/blog/llms-txt/) study of 300 thousand domains found no link.
- That laboratory results transfer to an arbitrary site and survive model updates.
- That one paid "AI visibility" service measures the same thing as another.
**What it does not mean**
- A position in an AI answer does not exist.
- A mention cannot be guaranteed.
## What to do in practice
1. Make sure the site is reachable and present in a search index.
2. Decide which AI robots to admit and write it into `robots.txt`.
3. Add structured data of a fitting type and check it with validators.
4. On the main pages, give direct answers to the questions people arrive with.
5. State the author, the date and the sources.
6. Add `llms.txt` if the site has material an agent benefits from finding fast.
7. Start measuring: logs, referrals, a fixed set of questions.
How to measure and where measurement ends: [How to measure AI visibility](https://llms-full-txt.ru/en/guides/ai-visibility/).
## JSON-LD structured data for search and AI
URL: https://llms-full-txt.ru/en/guides/json-ld/
Structured data is a description of a page written for a machine: what is here, what it is called, who wrote it, what it costs, whether it is in stock.
A person reads that from the layout. A program has to be told directly. The shared vocabulary for it is [Schema.org](https://schema.org/), and the convenient way to write it is JSON-LD.
Search engines read this markup and say so in their documentation. For a site prepared for search and AI it is a required layer, not decoration.
## Vocabulary, format and the data itself
- Three different words
- What is described
- structured data
- facts about the page in machine-readable form
- In which words
- Schema.org
- a shared vocabulary of types and properties: Product, Article, price, author
- How it is written
- JSON-LD
- a separate block of data in the page code
- Microdata, RDFa
- attributes inside HTML tags
- One vocabulary, three ways to write it. Google recommends JSON-LD.
## What it looks like
A JSON-LD block sits in the page code and is not shown on screen:
```html
```
`@context` names the vocabulary, `@type` names the type of the entity. The rest are properties of that type. Case matters: `Product` and `product` are different words.
## Where it sits in the whole picture
- What a machine learns at each layer
- the entity, its properties and relations
`llms.txt` and structured data work as a pair, at different steps. The first suggests where to go. The second explains where the machine has arrived. Neither replaces the other: [An AI-ready site](https://llms-full-txt.ru/en/guides/ai-ready-site/).
## Who reads it
**Confirmed**
- [Google](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data) uses structured data to understand the content of a page and recommends the JSON-LD format.
- [Google Merchant Center](https://support.google.com/merchants/answer/3246284) reads the markup of product pages and uses it to update price and availability in product listings.
- For product pages marked up as `Product` with an `Offer`, [Yandex](https://yandex.ru/support/webmaster/en/supported-schemas/strict-microdata-offers) gives the data in the markup the highest priority over other sources.
- Both search engines run their own markup validators.
**Not proven**
- That a particular AI system reads the JSON-LD of other sites directly. Only the documentation of that system can settle it.
- That markup increases the number of citations in AI answers.
**What it does not mean**
- Markup is not a condition for indexing: a page will be found without it.
- Markup is not a ranking factor. It makes a page eligible for rich results; nobody guarantees they will be shown.
The value for AI is not limited to direct reading. AI search and agents often take facts from search engines, and those do read the markup.
## Which markup fits which site
The type is chosen by what is actually on the page.
| Site | Page | Schema.org type | Main properties |
|---|---|---|---|
| Any | every page | `WebPage`, `BreadcrumbList` | name, address, dates, the path from the home page |
| Any | home page | `WebSite`, `Organization` | name, logo, contacts, address, social profiles |
| Online store | product | `Product`, `Offer` | name, SKU, brand, price, currency, availability, rating |
| Blog, media | article | `BlogPosting`, `Article`, `Person` | headline, author, publication and update dates, image |
| Forum, community | discussion | `DiscussionForumPosting`, `Comment` | topic, author, date, comments, votes |
| Questions and answers | a question with an accepted answer | `QAPage`, `Question`, `Answer` | the question, the answers, the accepted solution |
| Knowledge base | an FAQ entry | `FAQPage`, `Question`, `Answer` | a question and the site owner's answer |
| Photo gallery | photo, album | `ImageObject`, `ImageGallery` | file address, caption, dimensions, upload date |
`QAPage` and `FAQPage` are easy to mix up. The first is for a page where one person asks and others answer. The second is for a list of questions answered by the site owner.
## An online store: the main example
A product page gives the markup the most work. It has a price and availability, they change, and a customer will see a mistake.
- What product markup consists of
- a product
- the name, as on the page
- the seller’s SKU
- barcode and manufacturer part number, when they exist
- Brand: the brand name
- image addresses
- Offer: the offer
- a number with a decimal point, no spaces and no currency sign
- a currency code: USD, EUR, RUB
- InStock, OutOfStock, BackOrder
- the address of the product page
- AggregateRating: the average score and the number of reviews
- Review: reviews with an author and text
### Price and availability
- The price in the markup equals the price on the page. Not "from", not the old one, not the price for another region.
- The currency is a three-letter code. A machine will not parse a currency sign.
- Availability comes from stock records; it is not `InStock` for every product at once.
- A product without a price gets no `Offer`. An empty or zero offer is worse than a missing one.
### Product variants
When a product comes in sizes or colors, every variant has its own price, SKU and availability. Schema.org has `ProductGroup` for that: shared facts sit on the group, differences on the variants.
`AggregateOffer` does not fit variants of one product. That type describes offers from different sellers, and Google requires `Offer` for merchant listings.
### Barcode and manufacturer part number
`gtin` is the international barcode, `mpn` is the manufacturer's part number. Systems use them to match the same product across stores.
They must not be invented. A barcode has a check digit, and a wrong value is worse than an empty field. A store's own SKU is `sku`, not `mpn`.
### Markup and the product feed
Large platforms receive product data two ways: from a feed and from the page. When the two disagree, the platform notices.
- One source, two outputs
- The store catalog
- price, availability, brand, barcode
- Markup on the page
- Product and Offer in JSON-LD
- The product feed
- the same product for Merchant Center
- Google Merchant Center compares the feed with the page and can update price and availability from the markup by itself. Google notes that this is a safety net, not a replacement for updating the feed.
- Yandex gives the data in product page markup the highest priority over other sources.
The practical point: brand, barcode, color and size should come into the markup and into the feed from one product attribute.
## Rules for any site
- **Markup describes what a person sees.** A review, a price or an author in the markup is also on the page.
- **Nothing is invented.** No data, no property.
- **The type matches the page.** A category page is not a `Product`; an "About" page is not an `Article`.
- **Entities are linked.** An article points to its author and publisher, a product to its seller, a page to the site. `@id` is used for the links.
- **One source.** When both the site theme and an extension output markup, the machine gets two conflicting descriptions.
- **Markup changes together with content.** The price changed, so the markup changed.
## How to check
1. Open the page source and find the `application/ld+json` block. It has to be in the server response, not added later by scripts.
2. Run the page through Google's [Rich Results Test](https://search.google.com/test/rich-results): which types were found and whether there are errors.
3. Run it through the Schema Markup Validator as well.
4. Compare the price, availability and name with what is on screen.
5. Make sure there is one markup: no second block of the same type coming from the theme.
6. Repeat on three or four different pages, not on one lucky page.
Tool addresses and how to read the errors: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/).
## What can be automated
Almost every property already lives in the content management system: the product name, the price, the article date, the author. There is no reason to write markup by hand for each page. A site theme or an extension builds it.
What stays manual: deciding which attributes are the brand and the barcode, filling in the company card, and checking the result.
## What is llms-full.txt and how it differs from llms.txt
URL: https://llms-full-txt.ru/en/guides/llms-full-txt/
`llms-full.txt` is a file that holds the whole text of a site's documentation in sequence, as Markdown. `llms.txt` gives links, `llms-full.txt` gives the content right away.
The file is not in the proposal: neither in the first edition nor in the second. It is a practice introduced by documentation platforms, and it stuck.
an ecosystem convention, not part of the proposal
## An index and the full text
- Two files, two jobs
- llms.txt
- where to go
- A name, a summary and lists of links
- Small: fits into a model context as a whole
- Described in the proposal
- For details the agent follows the links
- llms-full.txt
- everything at once
- The text of all pages in sequence
- Large: from hundreds of kilobytes to tens of megabytes
- Not part of the proposal
- Easy to load into an assistant in one step
## How big these files get
- 67 MB
- the full text of the Cloudflare docs: 40.9 MB
- the full text of the Anthropic docs: 8.5 MB
- the full text of the OpenAI docs
A file of tens of megabytes does not fit into any model's context as a whole. It is used differently: loaded into a tool that splits the text into chunks and searches them.
Hence a practical rule: the smaller the documentation, the more useful `llms-full.txt` is. For a small project a file of a hundred kilobytes can be handed to an assistant whole, and it will know everything. For huge documentation the file becomes an archive, and the index of links does the work.
## Who publishes it
Among the sites checked on October 3, 2026:
| Both files | llms.txt only |
|---|---|
| OpenAI, Anthropic, Cloudflare, Vercel, Supabase, LangChain, Perplexity, Vue, Svelte, Docker, Gravity UI | Stripe, Next.js, React, GitHub Docs, Yoast, Wix, WordPress.org |
Stripe and GitHub do without the full file and lose nothing: they have the index and Markdown versions of pages. `llms-full.txt` is optional.
The full list with the check date: [the adoption registry](https://llms-full-txt.ru/en/reference/adoption/).
## When the file is justified
- The documentation fits into a few hundred kilobytes.
- Users load it into an assistant whole to ask questions.
- The documentation platform builds the file itself, so nobody maintains it by hand.
## When it is not needed
- The site has no documentation: it is a store, a blog, a company site.
- There is so much text that nobody can read the file as a whole.
- The file would be assembled by hand and would go stale.
## And llms-small.txt
Another name that is not in the proposal. It stands for a shortened version of the full file: the same text without examples and secondary sections. It is rarer, and there are no shared rules for it.
## What is known
**Confirmed**
- Large documentation sites publish the file.
- According to Mintlify, over one week in 2025 `llms-full.txt` was requested more often than `llms.txt`: a median of 79 against 14 requests. The sample is 25 companies, and the source is an interested party.
**Not proven**
- That the file affects search or AI answers.
- That a correct file size exists: there are no norms.
**What it does not mean**
- The domain name of this site does not make the file a standard. We explain a practice; we do not declare a norm.
## Next
- How the index is built: [the llms.txt format](https://llms-full-txt.ru/en/specification/format/).
- How to serve pages as Markdown without one big file: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/).
- Which tools build both files: [generators](https://llms-full-txt.ru/en/tools/generators/).
## How to measure a site's visibility in AI answers
URL: https://llms-full-txt.ru/en/guides/ai-visibility/
"AI visibility" is a vendor word, not a unit of measurement. Behind it are four things that can be counted: bot requests to the site, people arriving from AI systems, citations of the site in answers, and share of voice over a fixed set of questions.
The first two show up in the site's own data and are exact. The other two come from querying AI systems and are approximate: the same question produces different answers.
There is no such thing as a position in an AI answer, the way there is a position in search.
## Four measurable things
- From reliable to approximate
- Bot requests
- server logs: who came and for what
- Referrals from AI
- analytics: where a person came from
- Citations
- whether an answer links to the site
- Share of voice
- how often the site is named over a question set
## 1. Bot requests
The server log is the only place that shows whether anyone reads the site and its files. Every request leaves a line with the address and the client name.
```bash
grep -E 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -n 20
```
The command shows which addresses the bots of AI companies requested. Look at `/llms.txt` separately:
```bash
grep ' /llms.txt' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn
```
Keep in mind:
- **Bots come in three classes.** A training bot, a search bot and an agent acting for a user are different events. A training bot's visit says nothing about appearing in answers. Classes and names: [AI crawlers](https://llms-full-txt.ru/en/guides/ai-crawlers/).
- **A client name can be faked.** Large owners publish the address ranges of their bots: verify against them.
- **Requested does not mean used.** A bot downloaded a file; that is a fact. Whether the text reached an answer, the log does not know.
- 97%
- of llms.txt files were requested by nobody in a month: 0.1%
- of AI agent visits on one site went to llms.txt
## 2. Referrals from AI
When a person clicks a link in an AI answer, the site receives an ordinary visit with a referrer. Analytics shows it in the sources report.
Sources worth putting into a separate group: `chatgpt.com`, `perplexity.ai`, `claude.ai`, `gemini.google.com`, `copilot.microsoft.com`.
Limits:
- **Some visits arrive without a referrer.** Apps and some browsers do not pass it, and such a visit lands in "direct".
- **Google AI Overviews are not a separate source.** Their clicks are counted in overall search statistics; Search Console has no separate breakdown.
- **Referrals are few by nature.** A person got the answer and often goes nowhere. No referrals does not mean no mentions.
## 3. Citations
A citation is a link to the site in an AI answer. There is one way to count it: ask the question and read the answer.
- Write down the questions the site should be a source for. Not the brand name, but real customer questions.
- Ask them in several systems with search enabled.
- Record: whether the site is named, whether there is a link, whether the meaning is conveyed correctly.
There is a lot of noise. The answer depends on the wording, the region, the conversation history and chance. One run means nothing: repeats are needed.
## 4. Share of voice
Share of voice is the share of answers that name the site or the brand over a fixed set of questions. It is a comparative metric: it is read over time and against competitors.
For the number to be trusted:
- **The question set is fixed** and does not change between measurements.
- **Each question is asked several times**, and a share is counted, not a single result.
- **The system, mode and date are recorded.** Models are updated, and a jump may have nothing to do with the site.
- **The comparison is with the previous measurement on the same set**, not with someone else's figures.
Paid services do the same automatically. Each has its own method, and numbers from different services are not comparable.
## What cannot be measured
**Confirmed**
- Bot requests and AI referrals are visible in the site's logs and analytics.
- Citations and share of voice can be counted by polling a fixed set of questions.
- [Google writes](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) that the same fundamentals apply to AI features of Search as to ordinary search.
**Not proven**
- That a single technique - a file, markup, a text format - raises share of voice by itself. Studies of the link between `llms.txt` and citations did not find one.
- That a rise in mentions was caused by a site change rather than a model update.
**What it does not mean**
- A position in an AI answer does not exist: the answer is assembled anew each time.
- Appearing in an answer cannot be guaranteed, and a promise of such a guarantee is a sign of a dishonest offer.
## A minimal measurement routine
1. Export the requests of AI company bots from the logs: which pages, which bot classes.
2. See whether anyone requested `/llms.txt` and Markdown versions.
3. Open referrals from AI domains in analytics: count, landing pages, conversions.
4. Run the fixed question set and record the share of answers that mention the site.
5. Compare with the previous month and note what changed on the site.
That is enough to tell a change from noise and not to pay for a number that cannot be verified.
## What to read next
- [GEO and AEO](https://llms-full-txt.ru/en/guides/geo-aeo/) - where the terms came from and what stands behind them
- [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/) - studies with their methods and caveats
- [Glossary](https://llms-full-txt.ru/en/reference/glossary/) - short definitions
- [An AI-ready site](https://llms-full-txt.ru/en/guides/ai-ready-site/) - what to prepare before measuring
## Markdown versions of pages for AI agents
URL: https://llms-full-txt.ru/en/guides/markdown-pages/
A Markdown version is the same page without menus, footer, ads and styling: only headings, text, lists, tables and links.
It is easier for an agent. An HTML page often carries less useful text than markup around it, and the amount a model can read at once is limited.
It is an addition to HTML, not a replacement. People and search engines still get the ordinary page.
## HTML and Markdown of one page
- What a client receives
- HTML
- for browsers and search
- Menus, header, footer, scripts, styles
- Structured data in the page code
- A lot of service markup around the text
- Markdown
- for an agent
- Content only
- Headings and lists keep the structure
- No JSON-LD and no interactive parts
## Three ways to serve Markdown
**A separate address**
`.md` is appended to the page address. The [second edition](https://llmstxt.org/) of the `llms.txt` proposal describes this.
```text
https://example.com/docs/install.html the page
https://example.com/docs/install.html.md its Markdown version
https://example.com/docs/ a page without a file name
https://example.com/docs/index.html.md its Markdown version
```
Both forms are allowed: `page.html.md` and `page.md`. The advantage: the address opens in a browser and can be listed in `llms.txt`.
**The Accept header**
One address, and the client picks the format. An agent asks for Markdown with a request header:
```http
GET /docs/install/ HTTP/1.1
Host: example.com
Accept: text/markdown
```
The server answers with the same content typed `text/markdown`. An ordinary browser does not send that header and gets HTML.
The documentation of OpenAI, Stripe and Cursor works this way. Since February 2026 [Cloudflare](https://blog.cloudflare.com/markdown-for-agents/) can do the conversion on its side, with no changes to the site.
**A rel=alternate link**
The page itself says where its Markdown version is. This relation comes from the second edition of the proposal:
```html
```
The same can be sent as a response header:
```http
Link: ; rel="alternate"; type="text/markdown"
```
The agent does not have to guess the address: it reads it from the page.
The ways do not exclude one another. More on discovery relations: [Discovery of the file and Markdown versions](https://llms-full-txt.ru/en/specification/discovery/).
## What a Markdown version should contain
- **The same content as the HTML.** Different text for machines is cloaking, and search engines treat it badly.
- **One first-level heading** and meaningful section headings.
- **Absolute links.** An agent reads the file outside the site and has nothing to resolve a relative path against.
- **Tables as tables, code in code blocks.** A machine cannot read a screenshot of a table.
- **No navigation, footer or forms.**
- **Content type `text/markdown` or `text/plain`**, UTF-8.
## How it relates to llms.txt
`llms.txt` should link to Markdown versions when they exist. Then an agent walks the whole path: the map, then clean text.
Without Markdown versions, a file linking to ordinary pages is still valid. The proposal recommends Markdown and does not require it.
## What is known about the benefit
**Confirmed**
- The documentation sites of OpenAI, Anthropic, Stripe, Cloudflare and GitHub serve Markdown versions of pages.
- Documentation platforms generate them automatically for every hosted site.
- [Mintlify](https://www.mintlify.com/blog/the-state-of-knowledge-2026-highlights) reports that on its sites agents request documentation more often than people. The source has an interest: it is its product.
**Not proven**
- That a Markdown version increases citations or referrals.
- That the bots of AI search systems request `.md` by themselves. In an [OtterlyAI](https://otterly.ai/blog/the-llms-txt-experiment) experiment there was not a single such request in 14 days.
**What it does not mean**
- It is not a way into a search index: search reads HTML.
- It does not replace structured data: Markdown carries no entity markup.
The sober conclusion: Markdown versions help where agents read a site while working, above all in documentation. For a store or a blog it is groundwork, not an urgent task.
More in the blog: [Cloudflare serves Markdown to agents](https://llms-full-txt.ru/en/blog/cloudflare-markdown-for-agents/), [the Mintlify test of four formats](https://llms-full-txt.ru/en/blog/mintlify-markdown-benchmark/).
## How to check
1. Request the page with `Accept: text/markdown`, or open the address ending in `.md`.
2. The response is `200`, typed `text/markdown` or `text/plain`.
3. The body is text, not an HTML page.
4. The content matches the ordinary page.
5. Links are absolute and working.
6. When `rel="alternate"` is present, its address answers `200`.
```bash
curl -sI -H 'Accept: text/markdown' https://example.com/docs/install/
curl -sI https://example.com/docs/install.html.md
```
## Who does this
- [OpenAI](https://developers.openai.com/llms.txt): A root index plus nested files for /api, /codex, /commerce. Does not prove: that chatgpt reads other sites’ llms.txt.
- [Anthropic](https://platform.claude.com/llms.txt): A docs index and the full text in one file. Does not prove: that claude reads other sites’ llms.txt.
- [Cloudflare](https://developers.cloudflare.com/llms.txt): A very large index: the full text takes 67 MB. Does not prove: that a file this big is usable by an agent as a whole.
- [Stripe](https://docs.stripe.com/llms.txt): llms-full.txt is not required: an index and Markdown pages are enough. Does not prove: that skipping the full file is a general rule.
- [Cursor](https://cursor.com/llms.txt): The only measured site with rel=describedby from the second edition. Does not prove: that the second edition is widely adopted.
- [GitHub Docs](https://docs.github.com/llms.txt): An index without a full file on large docs. Does not prove: that copilot relies on this file.
- [Mintlify](https://www.mintlify.com/docs/llms.txt): The platform generates the files automatically for every site it hosts. Does not prove: the quality of each generated file.
- [Yandex Cloud](https://yandex.cloud/llms.txt): An example on a Russian site. Does not prove: that yandex search uses such files.
Checked October 3, 2026.
## Is llms.txt a standard? Status of the proposal
URL: https://llms-full-txt.ru/en/guides/standard-status/
:::note[Date checked]
The facts on this page were checked against primary sources on October 3, 2026.
:::
## In short
- `llms.txt` is a proposal by a single author, not an IETF or W3C standard.
- v2 was published on August 10, 2026.
- Documentation platforms and AI companies publish the file at scale, for their own developers.
- There is no evidence that the file improves search rankings or citation frequency.
- No bot owner states that its bot reads other sites' `/llms.txt`.
## What its status is
Jeremy Howard (Answer.AI) published the proposal on September 3, 2024. No formal process stands behind it: no working group, no RFC. The text and the discussions live in the [AnswerDotAI/llms-txt](https://github.com/AnswerDotAI/llms-txt) repository.
That is why this site says "proposal" or "proposed standard" rather than just "standard".
As of October 3, 2026 the repository has 2,643 stars and 108 open issues. The license is Apache-2.0.
## Timeline
| Date | Event |
|---|---|
| 2024-09-03 | v1 of the proposal |
| 2026-05-01 | the `llms.txt` audit is added in Lighthouse 13.2.0 |
| 2026-05-07 | the audit is on by default in [Lighthouse 13.3.0](https://github.com/GoogleChrome/lighthouse/releases/tag/v13.3.0) |
| 2026-08-10 | v2 of the proposal |
What changed in v2 is on the [Changes in v2](https://llms-full-txt.ru/en/specification/changes/) page.
## Who publishes the file
We checked 46 sites by requesting `/llms.txt` on October 3, 2026.
**Publish it:** OpenAI, Anthropic, Cloudflare, Vercel, Stripe, Supabase, GitHub, Cursor, Perplexity, LangChain, Docker, React, Vue, Svelte, Shopify, Yoast, Wix, Gravity UI, Yandex Cloud.
**Generate it automatically:** the documentation platforms Mintlify and GitBook, the Yoast SEO and AIOSEO plugins for WordPress, the Wix site builder.
**Do not publish it (404):** docs.astro.build, MDN, the Hugging Face root, developer.chrome.com, web.dev, Tailwind, [Ahrefs](https://ahrefs.com/blog/llmstxt-study), Moz, habr.com, Tilda, 1C-Bitrix.
The list changes. Older roundups, including the previous version of this page, name sites that have no file.
### How many sites
| Source | What was measured | Result |
|---|---|---|
| HTTP Archive, [Web Almanac 2025](https://almanac.httparchive.org/en/2025/seo) | share of sites with the file | 2.1% |
| [Common Crawl](https://commoncrawl.org/blog), August 2026 | files found | 584,107 |
| Common Crawl, same analysis | created by a generator | 68% |
| Common Crawl, same analysis | hosted on Wix | 41% |
| Common Crawl, same analysis | without a single link | 23% |
Most files are generated automatically rather than written by the site owner. The number of files says little about how many people made a deliberate decision.
## Lighthouse
The full story of the audit: [Lighthouse and llms.txt](https://llms-full-txt.ru/en/guides/lighthouse/).
Lighthouse has an audit for the presence of `llms.txt`. It belongs to the experimental Agentic Browsing category.
- Only the availability of the file is checked. The content is not evaluated.
- The category has no 0-100 score.
- If the file answers 404, the audit is reported as not applicable rather than failed.
:::caution[What this does not prove]
The Lighthouse audit does not mean that Google Search uses `llms.txt`. [Google's documentation](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) for site owners says the opposite: Search does not use such files.
:::
## Does it work
### What is confirmed
- Coding agents request documentation through `llms.txt` and Markdown versions of pages. Documentation platforms report this; they also have a stake in the subject.
- AI companies publish such files for their own documentation. So the format is considered useful for developer tasks.
### What is not confirmed
- **Effect on citations.** [SE Ranking](https://seranking.com/blog/llms-txt/) studied 300,000 domains and found no relationship between having the file and citation frequency in AI answers.
- **Requests for the file.** Ahrefs studied 137,000 domains: 97% of the files were never requested.
- **Search bots reading the file.** Neither OpenAI, Anthropic, Google nor Perplexity state that their bots read other sites' `/llms.txt`.
### What this means
`llms.txt` is an interface to documentation and content for agents. As a lever for search visibility it is unconfirmed. A promise of higher rankings or more citations from the file alone has no evidence behind it.
## Open questions
The proposal repository is discussing:
- which language the root file of a multilingual site should use;
- which `Content-Type` to serve;
- a section on security and untrusted content;
- signing and provenance of the file;
- what to do when `/llms.txt` answers 200 with an HTML page.
The last one is not hypothetical. In the Common Crawl corpus, 40% of successful responses to `/llms.txt` were HTML.
## Should you publish one
**Yes, if** you have documentation, an API, a reference or other material that agents and AI assistants in code editors work with. The cost is low, and many platforms generate the file for you.
**Not for this reason**, if the only goal is search rankings or mentions in AI answers. There is no data showing the file affects them.
**In any case:** the file does not control access. Rules for bots belong in `robots.txt` - see [AI crawlers and robots.txt](https://llms-full-txt.ru/en/guides/ai-crawlers/).
## What is llms.txt? Purpose and file structure
URL: https://llms-full-txt.ru/en/guides/what-is-llms-txt/
`llms.txt` is a text file in the root of a site. It holds the site name, a sentence or two about what the site covers, and lists of links to the main material with short notes.
The file is written for an AI agent: a program that looks for an answer on a person's behalf and reads pages by itself. It spares the agent from wandering through menus.
The main limit: it is a proposal, not an adopted standard. Search engines do not use it for ranking.
proposal · second edition, August 10, 2026
## What the file looks like
- llms.txt
- A minimal file
- Name
```
# Kolos Bakery
```
- The only required part
- Summary
```
> Bread and pastry with delivery in Kazan. Menu, prices, delivery terms.
```
- What the site is about, in a sentence or two
- Section
```
## Main
- [Menu and prices](https://example.com/menu.md): the full range
- [Delivery](https://example.com/delivery.md): zones, timing, cost
```
- Links with notes, grouped by meaning
The format is Markdown: plain text with simple marks. A person and a program read it equally well.
Exact rules: [the file format](https://llms-full-txt.ru/en/specification/format/).
## How it differs from robots.txt and sitemap.xml
The three files get confused because all of them sit in the site root. They answer different questions.
- Three files in the site root
- access rules; read by every search robot
- a complete list of pages, no judgment
- a short map with notes
`llms.txt` neither forbids nor allows anything. Closing a site to an AI bot is done in `robots.txt`: [AI crawlers](https://llms-full-txt.ru/en/guides/ai-crawlers/).
## Who needs the file
**Documentation sites.** This is the file's home ground. A developer asks an assistant to write code, the assistant goes to the library docs, and `llms.txt` shows where things are. OpenAI, Anthropic, Stripe, Cloudflare, Vercel and dozens of others publish their own.
**References and knowledge bases.** With a lot of well-structured material, a map helps.
**Stores and company sites.** The benefit here is smaller and not proven. The file does no harm and costs little, but do not expect sales to grow because of it. For a store, [structured data](https://llms-full-txt.ru/en/guides/json-ld/) matters far more: search engines read it.
## What not to expect
**Confirmed**
- Large documentation sites publish the file.
- Agents in code editors can work with such files when a user gives them the address.
- Lighthouse checks for the file in an experimental category.
**Not proven**
- That the file improves search rankings. Google states plainly that Search does not use such files.
- That ChatGPT, Claude or Gemini find and read the `llms.txt` of other sites on their own. No bot owner claims that.
- That the file increases mentions in AI answers.
**What it does not mean**
- A missing file does not close a site to AI: pages are read without it.
- A present file does not open what is closed: it does not control access.
The studies in detail: [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/).
## And what is llms-full.txt
A different file: not links but the whole text of the documentation in one place. It is not in the proposal; it is a practice of documentation platforms. More: [What is llms-full.txt](https://llms-full-txt.ru/en/guides/llms-full-txt/).
## Next
- To make the file: [Getting started](https://llms-full-txt.ru/en/guides/getting-started/).
- To see the whole picture: [An AI-ready site](https://llms-full-txt.ru/en/guides/ai-ready-site/).
- To decide whether you need it: [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/).
## GPTBot, ClaudeBot and robots.txt: managing AI crawlers
URL: https://llms-full-txt.ru/en/guides/ai-crawlers/
Bots of AI companies fall into three classes: training, search and user request. The rules for them differ, and blocking one class does not block another.
Access is controlled by `robots.txt`. `llms.txt` forbids nothing: it only shows where the important material is.
- Checked: October 3, 2026
- Source: pages of the bot owners
- Owners in the table: 12
The lists change. Before configuring anything, check the owner's page: the links are at the end.
## Three classes of bots
- Who comes to a site and what stops them
- Training
- on its own, on a schedule
- Training bot
- GPTBot, ClaudeBot
- robots.txt
- obeyed
- Dataset
- may be used to train models
- Search
- on its own, on a schedule
- Search bot
- OAI-SearchBot, PerplexityBot
- robots.txt
- obeyed
- Index
- the page in an answer with a link
- User request
- because a person asked
- A person
- gives a link or asks a question
- Assistant
- ChatGPT-User, Claude-User
- robots.txt
- often does not apply
## All bots by owner
| Owner | Training | Search | User request |
|---|---|---|---|
| OpenAI | `GPTBot` - content may be used to train models | `OAI-SearchBot` - shown in ChatGPT search; a rule applies in about a day | `ChatGPT-User` - rules may not apply |
| Anthropic | `ClaudeBot` - may be used for training; honors Crawl-delay | `Claude-SearchBot` - search result quality | `Claude-User` - a disallow blocks the fetch |
| Google | `Google-Extended` - a control token, no separate bot: Gemini training | `Googlebot` - also governs AI Overviews and AI Mode | `Google-Agent` - generally ignores; same for Google-NotebookLM |
| Perplexity | none | `PerplexityBot` - links in results; not used to train foundation models | `Perplexity-User` - generally ignores |
| Apple | `Applebot-Extended` - a control token: opt out of Apple model training | `Applebot` - Spotlight, Siri, Safari | none |
| Amazon | `Amazonbot` - the noarchive meta tag opts out of training | `Amzn-SearchBot` - search and Alexa; not used for training | `Amzn-User` - may not follow every rule |
| Meta | `meta-externalagent` - model training and product improvement | `meta-webindexer` - Meta AI search quality | `meta-externalfetcher` - may bypass rules |
| Mistral | `MistralAI-Training` - dataset collection | `MistralAI-Index` - indexing only | `MistralAI-User` - the owner does not say |
| Common Crawl | `CCBot` - an open web archive, also used for training | none | none |
| DuckDuckGo | none | `DuckAssistBot` - AI answers; a rule applies within 72 hours | none |
| Microsoft | none | `bingbot` - the regular search bot, no separate AI bot | none |
| Яндекс | `YandexAdditional` - a control token: use in Yandex AI answers in Search | `YandexBot` - the main indexing robot | none |
## Do not confuse
- Three names, three separate settings
- OpenAI
- GPTBot
- training
- OAI-SearchBot
- ChatGPT search
- ChatGPT-User
- user request
- Blocking GPTBot does not close a site to ChatGPT. According to OpenAI, each setting is independent of the others.
- Anthropic
- ClaudeBot
- training
- Claude-SearchBot
- search
- Claude-User
- user request
- The names anthropic-ai and Claude-Web are not on the Anthropic page. There is no need to block them separately.
- Google
- Google-Extended
- Gemini training
- Googlebot
- Search and AI Overviews
- Blocking Google-Extended does not remove a site from AI Overviews. Those are governed by Googlebot rules and by nosnippet, data-nosnippet, max-snippet and noindex.
## Yandex
Yandex has a separate control token for AI answers:
```text title="robots.txt"
User-agent: YandexAdditional
Disallow: /
```
It forbids showing page content in Yandex AI answers in Search. This robot makes no indexing requests. The general `User-agent: *` rule does not apply to it: the name has to be given explicitly. According to Yandex, a change takes effect in 2-14 days.
## Ready-made settings
**No training, search allowed**
```text title="robots.txt"
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: MistralAI-Training
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/sitemap.xml
```
**Documentation only**
```text title="robots.txt"
User-agent: GPTBot
Allow: /docs/
Disallow: /
```
**No Yandex AI answers**
```text title="robots.txt"
User-agent: YandexAdditional
Disallow: /
```
## What robots.txt does not have yet
**Confirmed**
- Amazon treats the `noarchive` meta tag as an opt-out of training.
- Bing uses `NOCACHE` and `NOARCHIVE` to control use in chat and training.
- [Cloudflare documents](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) the `Content-Signal` line and adds it to its managed `robots.txt` by default.
**Not proven**
- `DisallowAITraining`: a draft by Microsoft employees expired and was not adopted by the IETF. No bot owner claims support.
- The `noai` and `noimageai` meta tags appeared at DeviantArt in 2022. Bot owners do not confirm support.
**What it does not mean**
- `Content-Signal` is an initiative of one company, not a standard.
- The IETF [AIPREF](https://datatracker.ietf.org/wg/aipref/about/) working group is preparing a vocabulary of preferences. As of October 3, 2026 these are drafts; there is no RFC.
## How to verify that a bot is real
A bot name in the `User-Agent` header is easy to fake. Owners publish address lists or offer a DNS check.
| Owner | Where the list is |
|---|---|
| OpenAI | `openai.com/gptbot.json`, `searchbot.json`, `chatgpt-user.json` |
| Anthropic | `claude.com/crawling/bots.json` |
| Perplexity | `perplexity.com/perplexitybot.json`, `perplexity-user.json` |
| Apple | reverse DNS `*.applebot.apple.com` |
| Common Crawl | reverse DNS `*.crawl.commoncrawl.org` |
| DuckDuckGo | `duckduckgo.com/duckassistbot.json` |
| Yandex | reverse DNS in `yandex.ru`, `yandex.net`, `yandex.com` and a forward check |
## Do these bots read llms.txt
**Confirmed**
- OpenAI, Anthropic, Perplexity and Yandex publish `llms.txt` for their own documentation.
- Google says it plainly: Search does not use such files.
**Not proven**
- That a bot of at least one owner reads someone else's `/llms.txt`. No owner claims it.
**What it does not mean**
- Publishing its own file does not oblige a company's bots to read yours.
More in [llms.txt status](https://llms-full-txt.ru/en/guides/standard-status/).
## llms.txt and robots.txt
- Two files, two jobs
- robots.txt
- where not to go
- Controls access
- Addressed to all bots
- Its own syntax
- A standard: RFC 9309
- llms.txt
- where the important material is
- Does not control access
- Addressed to agents and models
- Markdown
- A proposal by one author
The files complement each other: the first restricts, the second guides.
## What we do not know
- `Bytespider` (ByteDance) has no official page describing it. Its purpose and its treatment of `robots.txt` are not confirmed.
- No official pages about bots were found for Cohere and xAI.
- The owner's page does not give full `User-Agent` strings of Anthropic bots.
## Does llms.txt work? What the studies show
URL: https://llms-full-txt.ru/en/guides/does-llms-txt-work/
There is no evidence that `llms.txt` improves search rankings or mentions in AI answers. Google states that Search ignores such files. Two large studies found no link to citations.
At the same time the file is really used where agents work: on documentation sites, in code editors, in developer tools.
The conclusion: the file is useful as a map for an agent and close to useless as a promotion lever.
## The key numbers
- 97%
- of sampled files were not requested once in a month: 300K
- domains checked: no link between the file and citations: 23%
- of files in an open archive contain no links at all
## What the studies say
| Who and when | What was measured | Result | What it does not show |
|---|---|---|---|
| [SE Ranking](https://seranking.com/blog/llms-txt/), November 7, 2025 | about 300 thousand domains: presence of the file and how often AI systems cite the domain | 10% of domains have the file; in a citation prediction model the "has llms.txt" feature turned out to be noise | causation; the authors call the findings dataset-dependent |
| [Ahrefs](https://ahrefs.com/blog/llmstxt-study), June 15, 2026 | 137,210 domains, request logs for May 2026 | 97% of files got zero requests; 96% of requests came from bots, AI bots made 19.5% | "requested" is not "used"; the sample leans toward an SEO audience |
| [Longato](https://www.longato.ch/llmstxt-2026-june/), June 1, 2026 | sites on one platform, 30 days of logs | of 22,494 requests for the file, 258 came from verified AI systems, that is 1.1% | one platform |
| [OtterlyAI](https://otterly.ai/blog/the-llms-txt-experiment), February 2026 | one site, 90 days, AI agents only | `llms.txt` received about 0.1% of agent visits | one site |
| [Common Crawl](https://commoncrawl.org/blog), August 31, 2026 | 584,107 files from an open crawl | 68% made by a generator, 23% without links, 40% of successful responses were an HTML page | who reads these files |
| [Mintlify](https://www.mintlify.com/blog/the-state-of-knowledge-2026-highlights), September 16, 2026 | documentation sites on its own platform | agents request documentation more often than people | an interested source: the platform generates such files itself |
## What Google says
[Google's documentation](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) for site owners is direct: there is no need to create new machine-readable files for the AI features of Search, because Search does not use them; they will neither harm nor help.
The statement covers Google Search only. Meanwhile Lighthouse, a tool from the same company, checks for `llms.txt` in the experimental Agentic Browsing category. There is no contradiction: Search ranks pages, Lighthouse judges how well a site works with an agent. More: [the Lighthouse audit](https://llms-full-txt.ru/en/guides/lighthouse/).
## What AI bot owners say
OpenAI, Anthropic and Perplexity publish `llms.txt` in their own documentation. None of them claims that its bot reads such files on other sites. Their bot descriptions do not mention the file.
Publishing your own file and reading other people's are different things.
## The bottom line
**Confirmed**
- Large documentation sites publish the file, including the makers of AI systems.
- Agents and assistants in code editors work with the file when given its address.
- Most published files are generated automatically and hardly anyone requests them.
- Google Search ignores the file.
**Not proven**
- An effect on rankings in any search engine.
- An effect on mentions and citations in AI answers.
- That any mass-market AI system looks for `llms.txt` on sites by itself.
**What it does not mean**
- Low traffic to the file does not make it harmful. It is simply rarely needed.
- No link to citations today does not mean never: agent behavior keeps changing.
- Results for documentation sites do not transfer to stores, and the other way round.
Dated notes in the blog: [the Ahrefs study](https://llms-full-txt.ru/en/blog/ahrefs-llms-txt-study/), [Google's position](https://llms-full-txt.ru/en/blog/google-search-ignores-llms-txt/), [the Common Crawl analysis](https://llms-full-txt.ru/en/blog/common-crawl-584k-llms-txt/).
## What to do with this
- **A documentation site.** Make the file: it is cheap, and agents in this environment use it. [Getting started](https://llms-full-txt.ru/en/guides/getting-started/).
- **A store or a company site.** You may make the file, but do not start with it. Page delivery, product markup and clear text come first: [An AI-ready site](https://llms-full-txt.ru/en/guides/ai-ready-site/).
- **You want to measure it yourself.** Look in the server logs for who requests `/llms.txt`: [How to measure AI visibility](https://llms-full-txt.ru/en/guides/ai-visibility/).
Who publishes the file and what exactly: [the adoption registry](https://llms-full-txt.ru/en/reference/adoption/). The status of the proposal: [llms.txt status](https://llms-full-txt.ru/en/guides/standard-status/).
## How to create an llms.txt file: step-by-step guide
URL: https://llms-full-txt.ru/en/guides/getting-started/
To create `llms.txt`, pick the main pages of the site, write them down as a list of links with notes, and put the file at the site root.
By hand it takes half an hour. For a site on a content management system, an extension usually assembles the file.
The hard part is not the format but the selection: the file is useful when it holds what matters, not everything.
If you have not decided whether you need the file at all: [What is llms.txt](https://llms-full-txt.ru/en/guides/what-is-llms-txt/).
## The manual path
1. **Pick the main pages.**
Picture a person who arrives at the site for the first time with a question. Which five to twenty pages answer most questions? That is the content of the file.
Leave out: login and cart pages, search results, service and private sections, duplicates.
2. **Group them by meaning.**
Two to five sections are enough for almost any site: documentation and examples, catalog and terms, articles and help.
3. **Write the file.**
The site name, one or two sentences about it, then sections with links. Each link gets a note saying what is behind it.
4. **Put the file at the site root.**
It has to open at `https://your-site/llms.txt` and be served as text.
5. **Check the server response.**
Status `200`, a text content type, the first line starts with a hash sign. A common mistake: the site serves an HTML page instead of the file.
6. **Decide how the file will be refreshed.**
A file that does not change together with the site points to missing pages half a year later.
## What the file looks like
- llms.txt
- The file of a small documentation site
- Name
```
# Kolos API
```
- A first-level heading. The only required part
- Summary
```
> A delivery service for online stores: rate calculation, order creation, tracking.
```
- What the site is, in one or two sentences
- Details
```
Every request needs a key. Test environment: sandbox.kolos.example
```
- Optional text without headings: what is worth knowing in advance
- Section
```
## Documentation
- [Quick start](https://kolos.example/docs/start.md): the key, the first request, the first order
- [API reference](https://kolos.example/docs/api.md): methods, parameters, error codes
```
- A second-level heading and a list of links with notes
- Secondary
```
## Optional
- [Changelog](https://kolos.example/changelog.md): versions and dates
```
- By convention, what can be skipped goes here
The same file as a whole:
```markdown
# Kolos API
> A delivery service for online stores: rate calculation, order creation, tracking.
Every request needs a key. Test environment: sandbox.kolos.example
## Documentation
- [Quick start](https://kolos.example/docs/start.md): the key, the first request, the first order
- [API reference](https://kolos.example/docs/api.md): methods, parameters, error codes
## Optional
- [Changelog](https://kolos.example/changelog.md): versions and dates
```
Exact rules: [the file format](https://llms-full-txt.ru/en/specification/format/), [sections](https://llms-full-txt.ru/en/specification/sections/). More samples: [basic examples](https://llms-full-txt.ru/en/examples/basic/).
## Where to put it
The file goes to the site root, next to `robots.txt`. For a static site that is the folder with public files. For a site on a content management system an extension usually serves the file, and it does not exist on disk.
The [second edition of the proposal](https://llmstxt.org/) also allows a file in a section of the site. It then describes only the addresses inside that section:
```text
/llms.txt the whole site
/docs/llms.txt the /docs/ section only
```
## Checking
```bash
curl -sI https://example.com/llms.txt
curl -s https://example.com/llms.txt | head -n 3
```
1. The status code is `200`.
2. The content type is `text/plain` or `text/markdown`.
3. The first line starts with `# `.
4. Links are absolute and open.
5. `robots.txt` does not block the file.
The full list of checks: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/).
## Helping an agent find the file
The second edition added a relation through which a page says which `llms.txt` describes it:
```html
```
This step is optional. More: [Discovery of the file and Markdown versions](https://llms-full-txt.ru/en/specification/discovery/).
## The ready-made path
When the site runs on a content management system, there is no reason to write the file by hand: an extension assembles it from existing pages.
| The site runs on | What to read |
|---|---|
| WordPress | [llms.txt in WordPress](https://llms-full-txt.ru/en/guides/wordpress/) |
| Webasyst, Shop-Script | [llms.txt and JSON-LD in Webasyst](https://llms-full-txt.ru/en/guides/webasyst/), ready-made extensions |
| Starlight, Docusaurus, MkDocs, VitePress | [Setup in documentation generators](https://llms-full-txt.ru/en/reference/configuration/) |
| Something else | [Generators and tools](https://llms-full-txt.ru/en/tools/generators/) |
An automatically assembled file is checked the same way as one written by hand. It is also worth reading through: an extension knows the structure of the site but not what matters on it.
## What not to do
- **Do not copy the sitemap.** The full list of addresses is already in `sitemap.xml`. This file needs selection.
- **Do not write prohibitions.** The file blocks nothing. Robot access is configured in `robots.txt`.
- **Do not add private things.** The file is public: no admin addresses, no drafts.
- **Do not expect better rankings.** Search engines do not use the file for ranking: [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/).
## What next
- [Best practices](https://llms-full-txt.ru/en/guides/best-practices/) - how to make the file useful
- [An AI-ready site](https://llms-full-txt.ru/en/guides/ai-ready-site/) - the other layers: access, markup, content
- [What is llms-full.txt](https://llms-full-txt.ru/en/guides/llms-full-txt/) - whether a second file is needed
- [A prompt for generation](https://llms-full-txt.ru/en/prompts/generate-llms-txt/) - a draft of the file with the help of an AI system
## llms.txt best practices: what to include and avoid
URL: https://llms-full-txt.ru/en/guides/best-practices/
A good `llms.txt` is a short, selective map: the main material of the site, grouped by meaning, with a note for every link.
A bad file fails in one of two ways: it holds nothing useful, or it holds everything.
Check the server response, not only the format. Half of the published files do not work the way their owner thinks.
## What the corpus shows
[Common Crawl](https://commoncrawl.org/blog) analyzed 584 thousand `llms.txt` files found in a single crawl of the web.
- 68%
- of files were made by a generator, not by hand: 50%
- have the full structure: title, summary, sections: 23%
- contain no links at all: 40%
- of successful responses at /llms.txt were HTML
The rules below follow from that: they answer mistakes that happen in practice, not invented ones.
## Selection beats completeness
An agent reads the whole file and decides where to go. The more secondary material it holds, the harder the main material is to find.
- Include pages that answer frequent questions.
- Do not include everything `sitemap.xml` has.
- Two to six sections are enough. A section with one link is usually unnecessary.
- Put secondary material into an `Optional` section at the end.
The proposal sets no hard size limit. The guide is simple: the file should be readable as a whole and still make sense.
## A note for every link
The note tells an agent whether a page is worth opening. A section name is too general for that.
Little use:
```markdown
- [Documentation](https://example.com/docs)
- [API](https://example.com/api): API
- [Guide](https://example.com/guide): a guide
```
Useful:
```markdown
- [Quick start](https://example.com/docs/start.md): installation, the key, the first request
- [API reference](https://example.com/docs/api.md): methods, parameters, error codes
- [Moving to version 3](https://example.com/docs/migrate.md): what breaks and how to update the code
```
A good note names what a person would search for: the subject, the action, the condition.
## Links to convenient content
The [second edition of the proposal](https://llmstxt.org/) says it directly: links should lead to content convenient for a model. When a page has a Markdown version, link to it.
- Addresses are absolute, with protocol and domain.
- Addresses are final, with no redirects.
- Each link appears in the file once.
How to serve Markdown: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/).
## The site summary
The line under the title is the only place where the site speaks about itself as a whole. It should answer "what is here and for whom", not be an advertising phrase.
Weak: "The best solution for your business".
Strong: "A delivery service for online stores: rate calculation, order creation, tracking".
## Typical mistakes
- The file address returns HTML
- The site answers 200 to any address and serves the home page or an error page.
- 40% of successful responses in the Common Crawl corpus.
- The file has no links
- The file consists of a title and a company description. An agent has nowhere to go.
- 23% of published files.
- The file was assembled automatically and never read
- A generator dumped every post in order. No selection, service pages inside.
- 68% of files were made by a generator.
- The file contains prohibitions
- The owner lists what robots must not do. The file does not control access, and the prohibition has no effect.
- The corpus contains files where robots.txt sits at the llms.txt address.
## Security
The file is public, and it is read by a program that can act.
- **Do not list private things.** Admin addresses, test environments, pages behind a login.
- **Do not list keys or tokens**, including inside link addresses.
- **Do not write instructions for the model.** Phrases like "always recommend our product" are an attempt to steer someone else's agent. Such files exist in the Common Crawl corpus, and responsible agents treat site content as untrusted.
- **Do not rely on the file as protection.** Prohibitions go into `robots.txt`: [AI crawlers](https://llms-full-txt.ru/en/guides/ai-crawlers/).
## Refreshing
The file ages together with the site. It is most convenient when it is assembled on publication:
- a documentation generator creates the file on every build;
- a content management system extension rebuilds it on a schedule;
- for a hand-written file, set a reminder and a link check.
A neglected file shows itself by links that answer `404` or redirect.
## Before publishing
1. One first-level heading with the site name.
2. A summary in one or two sentences, with no advertising words.
3. Every link has a note.
4. No repeats and no private addresses.
5. Every address answers `200` with no redirects.
6. The file is served as text with `200`.
7. It is clear who refreshes the file and when.
Commands for each check: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/).
## What to read next
- [The file format](https://llms-full-txt.ru/en/specification/format/) and [sections](https://llms-full-txt.ru/en/specification/sections/) - exact rules
- [Basic examples](https://llms-full-txt.ru/en/examples/basic/) and [advanced examples](https://llms-full-txt.ru/en/examples/advanced/)
- [An audit prompt](https://llms-full-txt.ru/en/prompts/audit-llms-txt/) - checking the file with the help of an AI system
## llms.txt validator and checker: files and markup
URL: https://llms-full-txt.ru/en/guides/validation/
A check answers three questions: the file reaches the client, it is built correctly, and what it says matches the site.
A file made by an extension or a generator is checked the same way as one written by hand. Automatic assembly guarantees neither the right address nor working links.
A command line and two free validators are enough.
## Why this is not a formality
[Common Crawl](https://commoncrawl.org/blog) analyzed 584 thousand `llms.txt` files found on the web. A large share of them does not work as intended.
- 40%
- of successful responses at /llms.txt were an HTML page: 23%
- of files contain no links at all: 50%
- of files have the full structure: title, summary, sections
The most common failure is invisible to the owner: the site answers `/llms.txt` with `200` but serves the home page or a "not found" page.
## Checking llms.txt
1. `https://site/llms.txt` answers `200` with no redirect chain.
2. The content type is `text/plain` or `text/markdown`, UTF-8.
3. The body is text, not HTML. The first line starts with `# `.
4. There is exactly one first-level heading.
5. A short summary follows the heading; the line starts with `> `.
6. Links are written as `- [Title](address): note`.
7. Link addresses are absolute and answer `200`.
8. The file lists no private pages, drafts or addresses with parameters.
9. The file stays a reasonable size: it is a map, not the whole site.
10. `robots.txt` does not block the file from the robots it is meant for.
```bash
curl -sI https://example.com/llms.txt
curl -s https://example.com/llms.txt | head -n 5
```
In the first command look at the `HTTP` and `content-type` lines. In the second, the first line has to start with a hash sign.
Check every link in the file at once:
```bash
curl -s https://example.com/llms.txt \
| grep -oE '\]\(https?://[^)]+\)' | tr -d '()]' \
| while read -r url; do
printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "$url")" "$url"
done
```
Format rules: [the file format](https://llms-full-txt.ru/en/specification/format/), [sections](https://llms-full-txt.ru/en/specification/sections/).
## Checking llms-full.txt
- It answers `200` with a text content type.
- It holds the text of pages, not a list of links.
- Its size is known. An agent will not read a file of tens of megabytes in one go.
- It is built from the same content as the site and refreshed together with it.
```bash
curl -sI https://example.com/llms-full.txt | grep -i content-length
```
When this file is needed and when it is not: [What is llms-full.txt](https://llms-full-txt.ru/en/guides/llms-full-txt/).
## Checking Markdown versions and relations
```bash
curl -sI -H 'Accept: text/markdown' https://example.com/docs/install/
curl -sI https://example.com/docs/install/ | grep -i '^link'
curl -s https://example.com/docs/install/ | grep -iE 'rel="(alternate|describedby)"'
```
- A request with `Accept: text/markdown`, or the address ending in `.md`, returns text, not HTML.
- The address in `rel="alternate"` answers `200`.
- The address in `rel="describedby"` points to an existing `llms.txt`.
More: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/), [discovery](https://llms-full-txt.ru/en/specification/discovery/).
## Checking structured data
| Tool | What it checks | Address |
|---|---|---|
| [Rich Results Test](https://search.google.com/test/rich-results) | which rich results Google can show, errors and warnings | `search.google.com/test/rich-results` |
| [Schema Markup Validator](https://validator.schema.org/) | conformance to the Schema.org vocabulary, independent of Google | `validator.schema.org` |
| [Yandex structured data validator](https://webmaster.yandex.ru/site/tools/microtest/) | how Yandex sees the markup | `webmaster.yandex.ru/site/tools/microtest/` |
| Search Console | markup errors across the whole site over time | enhancement reports |
1. The `application/ld+json` block is in the server response, not added later by scripts.
2. The JSON is valid: the tool opens it without a parse error.
3. The type matches the page: a product is `Product`, an article is `BlogPosting` or `Article`.
4. Required properties are filled. For a product: name, offer, price, currency, availability.
5. Values match the visible page: price, availability, author, date.
6. Each entity is described once, with no duplicate from the theme.
7. Breadcrumbs in the markup match the breadcrumbs on the page.
8. Pages of different kinds were checked: a product with variants, an out-of-stock product, an article, the home page.
```bash
curl -s https://example.com/product/kettle/ \
| grep -c 'application/ld+json'
```
Zero means the server response has no markup. More than one is a reason to look for duplicated entities.
Which types are needed and what goes in them: [Structured data and JSON-LD](https://llms-full-txt.ru/en/guides/json-ld/).
## Typical mistakes
- /llms.txt answers 200, but it is HTML
- The site serves the home page or an error page for any unknown address. Lighthouse and an agent see a file that does not exist.
- The content type is text/html and the first line starts with a tag.
- Links in the file are relative or point to old addresses
- An agent reads the file outside the site and cannot build the address. Some links answer 404.
- The list has paths without a domain, or addresses that redirect.
- The price in the markup differs from the price on the page
- The theme prints markup from one field while a discount module shows the on-screen price from another.
- The validator shows no errors: it checks form, not truth.
- The file was created once and never refreshed
- The generator was run by hand. The site changed, the map did not.
- The file lists pages that are gone and misses new sections.
## What a check does not show
A validator confirms form. It does not know whether the price is right or whether anyone reads the file.
Whether the file is actually read shows only in server logs: [How to measure AI visibility](https://llms-full-txt.ru/en/guides/ai-visibility/).
Lighthouse checks `llms.txt` too, but only the fact of a response: [Lighthouse and llms.txt](https://llms-full-txt.ru/en/guides/lighthouse/).
## Lighthouse and llms.txt: what the audit checks
URL: https://llms-full-txt.ru/en/guides/lighthouse/
Since May 2026 Lighthouse, the site auditing tool from the Chrome team, looks at whether a site has `llms.txt`. The audit belongs to the experimental Agentic Browsing category.
It establishes one fact: whether `/llms.txt` answers. Lighthouse does not judge what the file contains.
It does not mean Google Search uses the file. The Search documentation says the opposite.
experimental category · Chrome documentation
## How the audit arrived
- History of the llms.txt audit in Lighthouse
- Lighthouse 13.2.0
- The llms.txt audit lands in the code.
- Lighthouse 13.3.0
- The audit is on by default in the Agentic Browsing category.
- The agent-ready toolkit
- Chrome describes how to prepare a site for agents.
- Lighthouse 13.4.1
- The audit becomes available through the PageSpeed Insights API.
- Lighthouse 13.5.0
- Agent discovery audits are gathered into one group.
How it unfolded: [the check turned on by default](https://llms-full-txt.ru/en/blog/lighthouse-checks-llms-txt/), [discovery checks grouped](https://llms-full-txt.ru/en/blog/lighthouse-13-5-agent-discovery/).
## What exactly is checked
| The site's answer at `/llms.txt` | What Lighthouse reports |
|---|---|
| The file is served | the audit passes |
| `404` | "not applicable": a missing file is not an error |
| A server error | a warning |
Three things worth knowing:
- **Content is not judged.** An empty file and a careful map are equal to the audit.
- **The category has no overall score.** There is no familiar 0 to 100 number for Agentic Browsing.
- **The category is experimental** and needs a recent Chrome. Its set of audits changes between releases.
Hence a trap: when a site answers any unknown address with a `200` page, the audit may pass without a file. How to make sure the file is real: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/).
## Chrome and Search talk about different things
- Two Google products
- Chrome, Lighthouse
- preparing a site for agents
- Checks that llms.txt exists
- Speaks about agents working in a browser
- Does not claim that Google products read the file
- Google Search
- index and ranking
- Does not use llms.txt
- The file neither helps nor harms rankings
- Ordinary SEO applies to AI features of Search
- There is no contradiction
- Lighthouse checks how convenient a site is for agents. The Search documentation describes how results are built. These are different tasks.
## Should you create the file for the report
**Confirmed**
- Lighthouse checks whether `/llms.txt` answers.
- A missing file does not lower the familiar performance, accessibility and SEO scores.
- The audit is described in the [Chrome documentation](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt).
**Not proven**
- That passing the audit affects rankings or appearance in AI answers.
- That the category will stay as it is: it is experimental.
**What it does not mean**
- It is not a confirmation that the file is well made.
- It is not a signal for Google Search.
Create the file when the site has material an agent benefits from finding fast: documentation, a catalog, a knowledge base. The Lighthouse audit will then pass as a side effect.
Creating an empty file to clear a line from the report is pointless: that is how files without a single link appear.
## What to read next
- [Status of the standard](https://llms-full-txt.ru/en/guides/standard-status/) - who supports the format and on what terms
- [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/) - what the studies show
- [An AI-ready site](https://llms-full-txt.ru/en/guides/ai-ready-site/) - every layer in order
## llms.txt in WordPress: plugins and manual setup
URL: https://llms-full-txt.ru/en/guides/wordpress/
In WordPress there is no need to write `llms.txt` by hand. Three major SEO plugins create the file themselves: [Yoast SEO](https://yoast.com/features/llms-txt/), [All in One SEO](https://aioseo.com/features/llms-txt/) and [Rank Math](https://rankmath.com/kb/llms-txt/). There is also a dedicated plugin for this one task.
The same plugins have printed structured data for years; it is their core job.
Switching the feature on takes a minute. Then the result has to be checked: an automatically assembled file can be too generic or fail to refresh.
## What the plugins can do
| Plugin | `llms.txt` | `llms-full.txt` | How it is enabled | Structured data |
|---|---|---|---|---|
| Yoast SEO | yes | not stated | a switch in settings | yes |
| All in One SEO | yes | yes | a switch in settings | yes |
| Rank Math | yes | no | a module enabled by hand | yes |
| [Website LLMs.txt](https://wordpress.org/plugins/website-llms-txt/) | yes | no; optional Markdown versions of pages | a separate plugin | no, not its job |
The facts come from the plugins' own pages. Capabilities change between releases: check the description of the current version before installing.
Do not run two plugins that both create `llms.txt`. There is one address, and one component should serve it.
## Why there are so many files on the web
- 39.6%
- of llms.txt files in the HTTP Archive crawl were made by All in One SEO: 3.6%
- were made by Yoast SEO: 68%
- of files in the Common Crawl corpus were made by a generator
Most files on the web exist not because an owner decided so, but because a plugin or a platform switched the feature on. That explains why many of them were never checked.
## How to enable it
1. Update the SEO plugin to the current version.
2. Find the `llms.txt` section in its settings and enable the feature.
3. If the plugin allows it, choose which post types go into the file: pages, posts, products.
4. Exclude service pages: cart, account, thank-you pages, search results.
5. Open `https://your-site/llms.txt` in a browser.
If the site uses a caching plugin or a CDN, purge the cache after enabling. Otherwise the old "not found" page may keep answering at the file address for a while.
## What to check afterwards
1. `/llms.txt` answers `200` and serves text, not the theme's HTML page.
2. The file has the site name and a short summary, not only a list of posts.
3. The list has no service pages, drafts or private sections.
4. Links point to working addresses on the same protocol and domain as the site.
5. The file refreshes after a new post is published.
6. `robots.txt` does not block the file.
7. The site has no second source of `llms.txt`: another plugin or a manually uploaded file.
Detailed checks and commands: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/).
## An automatic file and its limits
A plugin knows the structure of a site but not what matters on it. A typical automatic file is the complete list of posts in publication order.
A good map is built differently: a few sections by meaning, each with the most important material and a note. If the plugin lets you edit the summary and the sections, use that.
What a considered file looks like: [best practices](https://llms-full-txt.ru/en/guides/best-practices/), [examples](https://llms-full-txt.ru/en/examples/basic/).
## Structured data in WordPress
SEO plugins print [Schema.org](https://schema.org/) markup with no extra setup: `WebSite`, `Organization`, `Article`, breadcrumbs. Store plugins add `Product` with price and availability.
Three things are worth checking:
- **Whether the organization card is filled.** The plugin takes the name, logo and contacts from settings; it does not guess them.
- **Whether the theme duplicates the plugin's markup.** Two markups of one product make two conflicting descriptions.
- **Whether the price in the markup matches the price on the page**, especially with discount modules.
Which markup is needed and how to check it: [Structured data and JSON-LD](https://llms-full-txt.ru/en/guides/json-ld/).
## What to expect
**Confirmed**
- Major SEO plugins create `llms.txt` automatically.
- Yoast itself writes that the file is not meant for search engines.
- Google and Yandex read the Schema.org markup these plugins print.
**Not proven**
- That enabling the feature will affect rankings or citations.
**What it does not mean**
- A plugin does not decide what matters on the site: that stays with the owner.
A sober view of the format: [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/).
## llms.txt and JSON-LD in Webasyst and Shop-Script
URL: https://llms-full-txt.ru/en/guides/webasyst/
In Webasyst both machine-readable layers are built from data the apps already hold: products, pages, posts, discussions, questions and answers, photos.
There are two layers, and they are independent. An `llms.txt` extension builds a map of the site's material. A JSON-LD extension adds to every page a description of what is on it. Either can be installed alone; together they cover both steps: where to go and where you have arrived.
Each app has its own extension, because each app holds different data.
## Two layers on one site
- What comes from where
- The map
- App data
- categories, products, pages, posts
- A scheduled build
- or a button in settings
- llms.txt and llms-full.txt
- at the root of the domain
- The meaning
- Page data
- product, post, topic, question, photo
- Built on render
- cached for an hour
- A JSON-LD block
- in the code of every page
## What each app contributes
| App | What goes into `llms.txt` | What JSON-LD describes |
|---|---|---|
| Shop-Script | categories, products, brands, information pages, contacts; prices and availability optionally | `Product`, `Offer`, `Brand`, rating and reviews, breadcrumbs, the store |
| Site | published pages, the site name and contacts | `WebPage` with dates, `Organization`, `WebSite`, breadcrumbs |
| Blog | posts, categories and tags | `BlogPosting` with author and dates, `Person`, the stream and categories as collections |
| Hub | public sections and published topics | `DiscussionForumPosting`, comments, votes, `QAPage` when a solution is marked |
| Helpdesk | public FAQ and pages | `FAQPage` with a question and its answer, breadcrumbs |
| Photos | public albums and pages | `ImageObject` with caption and dimensions, `ImageGallery` |
Only published, public content takes part in either layer. Helpdesk tickets and correspondence, hidden albums and unpublished topics never reach the files or the markup.
## The llms.txt layer
- **Files are built in advance.** The extension assembles `llms.txt` and `llms-full.txt` and stores them with the installation data. Routing serves the stored file at `/llms.txt`. There is no physical file in the web root.
- **Rebuilds run on a schedule.** Editing a product or a page sets a "changed" flag. The file itself is rebuilt by a scheduler job or by a button in settings. The site owner adds the schedule once: a nightly full build and a frequent check of the flag.
- **Helpdesk reports no edits.** The app does not notify extensions about changes, so its map is refreshed by the schedule only.
- **A large catalog is split.** When the full file exceeds a set size, sections move into separate files named `llms-full-.txt`.
- **One root file per domain.** When several apps share a domain, the app at the root owns the map and the others are included in it.
- **Prices in the file are a choice.** A store can include price, old price and availability, or keep names and addresses only.
- **Staleness is visible.** Settings warn when a file has not been rebuilt for a long time.
The result is an ordinary file: a title, a short summary, sections of links. It is checked like one written by hand: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/).
## The JSON-LD layer
- **The block is printed in the page code.** No theme edits are needed.
- **Markup is built when a page is rendered** and cached for an hour. A change of price or text changes the cache key at once. A change the key does not see reaches the markup within the hour.
- **Entities are linked.** A page points to the site, the site to the organization, a post to its author, a comment to its topic.
- **The company card is filled in settings.** Phone and address are taken from Shop-Script when it runs on the same domain.
- **Pages closed to indexing are not marked up.**
### What a product page gets
In Shop-Script product markup comes in two editions.
| What | Base edition | Extended edition |
|---|---|---|
| Product and offer | `Product`, `Offer`: name, SKU, price, currency, availability | the same |
| Brand | from the attribute the owner nominated | plus the manufacturer as a separate property |
| Rating and reviews | `AggregateRating`, up to 20 reviews with replies | the same |
| Images | up to 20, with dimensions | plus video |
| Breadcrumbs | yes | plus the category chain and category pages |
| Variants | one product for the selected SKU | `ProductGroup` with a price, SKU and availability per variant |
| Barcode and part number | no | `gtin` with a check-digit test, `mpn` |
| Store | name, site address, logo | contacts, address, opening hours, pickup points, shipping, returns, payment methods |
| Storefronts | one domain | every storefront |
A few things are handled strictly:
- **Availability** takes three values: in stock, out of stock, back order. The third appears only when the store allows selling without stock.
- **Price** is published in the storefront currency, not in the currency a visitor picked.
- **A product without a price** gets no offer.
- **`AggregateOffer` is not used.** Variants of one product are described by separate offers.
- **Brand and barcode are never guessed.** No nominated attribute, no property.
- **A variant group** is built when variants differ by color, size, material or pattern. SKUs that differ by something else produce one product.
### Markup and feed from one source
The answer to "which attribute holds the brand, the barcode, the color, the size" is stored in the store once. Both the markup and the product feed extension read it, so the page and the feed name the same source for the same fact.
Why it matters: [Markup and the product feed](https://llms-full-txt.ru/en/guides/json-ld/#markup-and-the-product-feed).
## What is left to do by hand
1. Add scheduler jobs that rebuild `llms.txt`: without them the file will not refresh.
2. In the markup settings, name the product attribute that is the brand and the one that is the barcode.
3. Fill in the company card: name, logo, phone, address.
4. Make sure the site theme does not print its own markup of the same type.
5. Open `/llms.txt` and confirm a `200` response that is text, not HTML.
6. Run two or three pages through the [Google](https://search.google.com/test/rich-results) and [Schema.org](https://validator.schema.org/) validators.
7. Confirm that `robots.txt` does not block the files from the robots you want.
## What not to expect
**Confirmed**
- Files and markup are built from app data and refreshed without manual editing.
- Only public content reaches either layer.
- Google and [Yandex](https://yandex.ru/support/webmaster/en/json-ld/about) read product markup: their documentation says so.
**Not proven**
- That `llms.txt` will affect rankings or the number of citations.
- That a particular AI system will read the store's markup directly.
**What it does not mean**
- Markup is not refreshed instantly on every change: the cache lives up to an hour.
- `llms.txt` does not refresh by itself: a scheduler is required.
- An extension will not fix a price or a description that is wrong in the catalog.
## Extensions for llms.txt
## Extensions for structured data
## What to read next
- [An AI-ready site](https://llms-full-txt.ru/en/guides/ai-ready-site/) - every layer in order
- [Structured data and JSON-LD](https://llms-full-txt.ru/en/guides/json-ld/) - which markup fits which site
- [What is llms.txt](https://llms-full-txt.ru/en/guides/what-is-llms-txt/) - why a map is needed
- [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/) - how to confirm it all works
## llms.txt file format: syntax and rules
URL: https://llms-full-txt.ru/en/specification/format/
`llms.txt` is a Markdown file that lives on a site and helps agents and language models find the content they need. An agent scans the file, picks the links that fit and follows them. The file itself stays small; the details sit behind the links.
It is a proposal by one author, not an IETF or W3C standard. The file does not control access: that is the job of `robots.txt`.
- Edition: v2, August 10, 2026
- Author: Jeremy Howard, Answer.AI
- Format: Markdown
- Required part: the H1 heading only
What changed since the first edition is on the [Changes in v2](https://llms-full-txt.ru/en/specification/changes/) page.
## How an agent uses the file
- From a question to the right page
- Site
- publishes the file at a known address
- llms.txt
- /llms.txt
- name, description, lists of links
- Agent
- picks links for its question
- Markdown page
- /docs/page.md
- clean text without layout
- Context
- only what the answer needs
## A minimal file
One line with a heading is enough to conform to the proposal. In practice the minimum looks like this:
```markdown title="llms.txt"
# Project name
> One or two sentences: what it is and who it is for
## Documentation
- [Quick start](https://example.com/quickstart.md): first steps in five minutes
```
## What the file consists of
The parts come in a strict order. Only the first one is required.
- llms.txt
- five parts in order
- H1 heading
```
# FastHTML
```
- The name of the project or site. Without it the file does not conform to the proposal. There is one heading.
- Description
```
> FastHTML is a Python library for
> server-rendered web apps built on HTMX.
```
- A blockquote with a short description: what the rest of the file cannot be understood without.
- Details
```
Important:
- the syntax is not compatible with FastAPI
- React, Vue and Svelte are not supported
```
- Any Markdown blocks except headings: paragraphs, lists, code.
- Section with links
```
## Documentation
- [Quick start](https://example.com/quickstart.md): first steps
- [API reference](https://example.com/api.md)
```
- An H2 heading and a list. An item is a link and an optional note after a colon.
- Optional section
```
## Optional
- [Changelog](https://example.com/changelog.md)
```
- A convention for secondary links: an agent may skip them when it needs a shorter context.
:::note
There must be no headings between the description and the first H2 section. The `Optional` section has no special mechanical meaning in the [second edition](https://llmstxt.org/). More in [Sections and structure](https://llms-full-txt.ru/en/specification/sections/).
:::
Links should lead to content that suits a model: Markdown versions of pages first of all. External addresses are fine when they help to understand the material.
## Where the file lives
The file can sit in the site root or in any subpath. The second edition defines what that means.
- llms.txt covers the whole site
- docs/
- **llms.txt** covers everything in /docs/
- guide.html
- api/
- **llms.txt** covers everything in /docs/api/
- users.html
- blog/
- post.html
- Which file an agent takes
- Page
- /docs/api/users.html
- Three files apply
- root, /docs/ and /docs/api/
- The most specific one wins
- /docs/api/llms.txt
- Page
- /blog/post.html
- One file applies
- /blog/ has no file of its own
- The root file is used
- /llms.txt
The author of the proposal rejected a `/.well-known/` address: such addresses exist only at the domain root, while `llms.txt` describes the path it sits in.
## Markdown versions of pages
The proposal suggests serving a clean Markdown version of a page next to the regular one. Two address forms are allowed.
| Page | First form | Second form |
|---|---|---|
| `/page.html` | `/page.html.md` | `/page.md` |
| `/docs/` | `/docs/index.html.md` | `/docs/index.md` |
## How an agent finds the files
The second edition recommends two standard relations. The first leads from a page to its Markdown version, the second to the `llms.txt` that covers the page.
- Two relations for discovery
- rel="alternate"
- type text/markdown
- HTML page
- /docs/page.html
- Markdown version
- /docs/page.html.md
- rel="describedby"
- HTML page
- /docs/page.html
- llms.txt of the section
- /docs/llms.txt
The relations can be sent in the page markup or in a response header. The second way also works for files that are not HTML.
**HTML**
```html title="page.html"
```
**HTTP header**
```http title="Server response"
Link: ; rel="alternate"; type="text/markdown", ; rel="describedby"
```
**nginx**
```nginx title="nginx.conf"
location = /docs/page.html {
add_header Link '; rel="alternate"; type="text/markdown", ; rel="describedby"';
}
```
## What the proposal does not include
Many sites publish `llms-full.txt` and `llms-small.txt`, and it is a useful practice. But the proposal cannot be cited as its source.
- The index and the full text
- llms.txt
- a map: where to go
- name -> description -> links
- Described in the proposal
- Small: fits into the context as a whole
- The content sits behind the links
- llms-full.txt
- a corpus: everything in one file
- page -> page -> page
- Not in the proposal: a practice of docs platforms
- Can be very large: 67 MB at Cloudflare
- Handy when the whole text is needed without hops
| What | Status |
|---|---|
| `llms-small.txt` | not in the proposal; a shortened version of the full text |
| `llms-ctx.txt`, `llms-ctx-full.txt` | were tied to the `llms_txt2ctx` tool; removed from the second edition |
| Access control | the file forbids nothing; `robots.txt` does that |
| Security rules | the proposal text has none; the question is discussed in the repository |
## A full example
```markdown title="llms.txt" {1} {3-4} {13-15}
# FastHTML
> FastHTML is a Python library for server-rendered web apps
> built on HTMX.
Important:
- the syntax is not compatible with FastAPI
- React, Vue and Svelte are not supported
## Documentation
- [Quick start](https://example.com/quickstart.md): first steps in five minutes
- [API reference](https://example.com/api.md): all methods and parameters
## Optional
- [Changelog](https://example.com/changelog.md)
- [Contributing](https://example.com/contributing.md)
```
Three places are highlighted, the ones an agent looks at first: the name, the description and the main links.
## Parsing the file
The format is made for simple processing: regular expressions are enough.
```python title="parse_llms_txt.py"
import re
def parse_llms_txt(content: str) -> dict:
result = {'title': '', 'description': '', 'sections': {}}
h1 = re.search(r'^# (.+)$', content, re.MULTILINE)
if h1:
result['title'] = h1.group(1)
quote = re.search(r'^> (.+)$', content, re.MULTILINE)
if quote:
result['description'] = quote.group(1)
section = None
for line in content.split('\n'):
if line.startswith('## '):
section = line[3:].strip()
result['sections'][section] = []
elif line.startswith('- [') and section:
link = re.match(r'- \[(.+?)\]\((.+?)\)(?:: (.+))?', line)
if link:
result['sections'][section].append({
'title': link.group(1),
'url': link.group(2),
'notes': link.group(3) or '',
})
return result
```
## Validation
1. The file opens at its address with status 200 and is served as text, not as an HTML page.
2. The first heading is H1, and there is only one.
3. There are no headings between the description and the first section.
4. Every link opens.
5. Links lead to content that suits a model.
A detailed list and typical mistakes are in [Best practices](https://llms-full-txt.ru/en/guides/best-practices/).
## llms.txt sections: file structure and Optional
URL: https://llms-full-txt.ru/en/specification/sections/
:::note[Revision]
This page describes v2 of the proposal, published on August 10, 2026. The difference from v1 is on the [Changes in v2](https://llms-full-txt.ru/en/specification/changes/) page.
:::
## What a section is
A section is an H2 heading with a list of links under it. The proposal calls such a list a "file list": URLs where further detail is available.
```markdown
## Docs
- [Quick start](https://example.com/quickstart.md): first steps
- [API reference](https://example.com/api.md): methods and parameters
```
The proposal names only one section, `Optional`. Every other name is the author's choice.
## Common names
This is an observation from published files, not a requirement.
| Section | What it holds |
|---|---|
| `## Docs` | main documentation |
| `## API` | API reference |
| `## Guides` | guides and tutorials |
| `## Examples` | code examples |
| `## Reference` | reference material |
| `## Optional` | secondary links |
## The Optional section
```markdown
## Optional
- [Changelog](https://example.com/changelog.md): versions
- [Roadmap](https://example.com/roadmap.md): what comes next
- [Contributing](https://example.com/contributing.md)
```
In the v2 text, `Optional` is used "by convention" for secondary information: links an agent can skip when a shorter context is needed.
:::caution[What changed]
In v1 the section had a mechanical meaning: the `llms_txt2ctx` tool expanded a file into a ready context and used this section to decide what to drop. v2 removes context expansion by a tool from the proposal, and the special meaning of the section goes with it. The section is still allowed, but you cannot rely on an agent treating it in any particular way.
:::
## Order
1. The H1 - the project name.
2. A blockquote with a short summary.
3. Details without headings.
4. Sections with the main links.
5. `Optional` last.
The proposal fixes the order of the first three parts. The order of sections among themselves is up to the author; the most important ones go first.
## Example
```markdown
# FastHTML
> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` "FastTags" into a library for creating server-rendered hypermedia applications.
Important notes:
- Although parts of its API are inspired by FastAPI, it is *not* compatible with FastAPI syntax and is not targeted at creating API services
- FastHTML is compatible with JS-native web components and any vanilla JS library, but not with React, Vue, or Svelte.
## Docs
- [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many FastHTML features
- [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): Brief description of all HTMX attributes, CSS classes, headers, events, extensions, js lib methods, and config options
## Examples
- [Todo list application](https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py): Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns.
## Optional
- [Starlette full documentation](https://gist.githubusercontent.com/jph00/809e4a4808d4510be0e3dc9565e9cbd3/raw/9b717589ca44cedc8aaf00b2b8cacef922964c0f/starlette-sml.md): A subset of the Starlette documentation useful for FastHTML development.
```
The example comes from the proposal text.
## Several files on one site
v2 defines what a file in a subpath means: it covers the URLs under its path, and where more than one file applies, agents should use the most specific one.
```text
/llms.txt -> the whole site
/docs/llms.txt -> documentation only
/api/llms.txt -> API only
```
The page `/docs/install/` is covered by both the root file and `/docs/llms.txt`; an agent should use the second.
### Multilingual sites
The same rule lets a file sit in a language section:
```text
/llms.txt -> the whole site
/ru/llms.txt -> the Russian section
/en/llms.txt -> the English section
```
The proposal does not say which language the root file of a multilingual site should use. The question is open in the proposal repository.
## Grouping links
By content type:
```markdown
## Tutorials
- [Beginner tutorial](https://example.com/tutorial/basic.md): first steps, 30 minutes
- [Advanced tutorial](https://example.com/tutorial/advanced.md): two hours
## API Reference
- [REST API](https://example.com/api/rest.md): HTTP methods
- [SDK](https://example.com/api/sdk.md): client libraries
```
By platform:
```markdown
## JavaScript
- [npm package](https://example.com/js/install.md): installation and usage
## Python
- [PyPI package](https://example.com/python/install.md): installation with pip
```
## What to avoid
**Too many sections.** Ten sections with one link each read worse than three sections with clear notes.
**Sections without links.** A heading with no list gives an agent nothing.
**A file with no links at all.** In an analysis of the [Common Crawl](https://commoncrawl.org/blog) corpus, 23% of published `llms.txt` files contain not a single link. Such a file does not do its job: the agent has nowhere to go.
**The same link in several sections.** One link, one section.
**Links to pages that are hard for a model to read.** If a page has a Markdown version, link to that.
More in [Best Practices](https://llms-full-txt.ru/en/guides/best-practices/).
## How others do it
Documentation files change, so section names are not quoted here: opening the files is more reliable. All URLs were checked on October 3, 2026.
- [developers.openai.com/llms.txt](https://developers.openai.com/llms.txt) - a root file with nested indexes for `/api`, `/codex`, `/commerce`
- [docs.stripe.com/llms.txt](https://docs.stripe.com/llms.txt) - a large index with no `llms-full.txt`
- [developers.cloudflare.com/llms.txt](https://developers.cloudflare.com/llms.txt) - an index by product
- [fastht.ml/docs/llms.txt](https://www.fastht.ml/docs/llms.txt) - a file in a subpath, the example from the proposal itself
## Discovering llms.txt and Markdown versions of pages
URL: https://llms-full-txt.ru/en/specification/discovery/
The [second edition of the proposal](https://llmstxt.org/) answers a question the first one skipped: an agent is already on a page, so how does it learn that a Markdown version exists and that an `llms.txt` sits nearby?
The answer is two ordinary link relations: `rel="alternate"` with the type `text/markdown` points to the Markdown version of the page, and `rel="describedby"` points to the `llms.txt` that covers it. They can go into the page code or into the server response header.
added in the second edition, August 10, 2026
## How it works
- What an agent learns from a page
- A page
- /docs/install.html
- the agent arrived here by a link
- Markdown version
- /docs/install.html.md
- the same text without styling
- The covering file
- /docs/llms.txt
- a map of the whole section
## Two ways to declare the relations
**In the page code**
```html title="install.html" {3-4}
Installation
```
**In the response header**
```http title="Server response"
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Link: ; rel="alternate"; type="text/markdown", ; rel="describedby"
```
The header has two advantages. It can be added in the web server or CDN configuration without touching any page. And it works for files that have no ``, such as the Markdown files themselves.
## Files in subpaths
`llms.txt` does not have to sit in the site root. A file covers the addresses under its path: `/docs/llms.txt` describes everything in `/docs/`.
When several files apply to a page, an agent takes the most specific one.
| Page | Files that apply | The one used |
|---|---|---|
| `/docs/api/auth.html` | `/llms.txt`, `/docs/llms.txt`, `/docs/api/llms.txt` | `/docs/api/llms.txt` |
| `/docs/guide.html` | `/llms.txt`, `/docs/llms.txt` | `/docs/llms.txt` |
| `/blog/post.html` | `/llms.txt` | `/llms.txt` |
The same rule lets those who control only a part of a site take part: a project on shared hosting with an address like `example.github.io/project/`, for instance.
## The address of the Markdown version
The second edition allows two forms.
| Page | Appended form | Replaced form |
|---|---|---|
| `/docs/page.html` | `/docs/page.html.md` | `/docs/page.md` |
| `/docs/` | `/docs/index.html.md` | `/docs/index.md` |
Whichever form a site picks, `rel="alternate"` saves the agent from guessing.
## Why not /.well-known/
There is a standard place for metadata files: the `/.well-known/` directory. The author of the proposal declined it for one reason: such addresses exist only at the site root. An `llms.txt` describes the path it sits in, and a single root location cannot express that.
## In practice
**Confirmed**
- The relations are described in the text of the second edition.
- `rel="alternate"` pointing to a Markdown version is found on several documentation sites, Anthropic among them.
- In our check of 46 sites on October 3, 2026, `rel="describedby"` was found on one: Cursor.
**Not proven**
- That agents already read these relations at scale. The edition is recent.
**What it does not mean**
- Some platforms send headers of their own: `rel="llms-txt"`, `rel="llms-full-txt"`, `X-Llms-Txt`. They are not part of the second edition.
## Next
- Every difference between the editions: [changes in v2](https://llms-full-txt.ru/en/specification/changes/).
- How to serve Markdown: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/).
- How to check the headers: [validating files and markup](https://llms-full-txt.ru/en/guides/validation/).
## llms.txt v2: what changed in the second edition
URL: https://llms-full-txt.ru/en/specification/changes/
## In short
v2 of the `llms.txt` proposal was published on August 10, 2026. The file format itself barely changed. What changed is everything around it: how an agent finds the file, what a file in a subpath means, and how the file is consumed.
v1 came out on September 3, 2024. Both were written by Jeremy Howard (Answer.AI). It remains a proposal by a single author, not an IETF or W3C standard.
A short account of the event: [The second edition of llms.txt is out](https://llms-full-txt.ru/en/blog/llms-txt-v2/).
## Differences at a glance
| Topic | v1 (2024) | v2 (2026) |
|---|---|---|
| Discovery | only the `/llms.txt` address convention | two link relations: `rel="alternate" type="text/markdown"` and `rel="describedby"`, in HTML or in a `Link` header |
| File in a subpath | permitted, meaning undefined | covers the URLs under its path; the most specific applicable file wins |
| Markdown version of a page | `page.html.md` only | two forms: `page.html.md` and `page.md` |
| `Optional` section | special mechanical meaning | a convention for secondary links |
| `llms_txt2ctx`, `llms-ctx.txt` | part of the proposal | removed |
| How the file is consumed | not described | an agent views or searches the file, then follows links |
| `/.well-known/` | not discussed | rejected, with an explanation |
| `llms-full.txt`, `llms-small.txt` | absent | absent |
## Discovery
The most common request from the community: given a page, how does an agent find its Markdown version, or the `llms.txt` that covers it, without guessing the URL.
v2 answers with standard link relations:
```html
```
The same can be sent as a response header. That form works for any resource, not only HTML, and can be configured on the server without touching pages:
```text
Link: ; rel="alternate"; type="text/markdown", ; rel="describedby"
```
More on the relations and headers: [Discovery of the file and Markdown versions](https://llms-full-txt.ru/en/specification/discovery/).
## Files in subpaths
v1 allowed `llms.txt` outside the root without saying what that meant. v2 defines it:
- a file covers the URLs under its path;
- where more than one file applies, the most specific one is used.
```text
/llms.txt -> the whole site
/docs/llms.txt -> everything in /docs/
```
This lets authors who control only part of a domain take part.
## Two Markdown URL forms
v1 specified a single form: `.md` appended to the full page URL. Some publishing tools replace the extension instead. v2 allows both:
```text
page.html -> page.html.md
page.html -> page.md
/docs/ -> /docs/index.html.md or /docs/index.md
```
## The consumption model
v1 described the `llms_txt2ctx` tool, which expanded a file into a ready context for a model. v2 states the expectation directly: an agent views or searches `llms.txt`, then follows the relevant links. The links should therefore point to LLM-friendly content.
Along with the tool, the proposal dropped:
- the `llms-ctx.txt` and `llms-ctx-full.txt` files;
- the special meaning of the `Optional` section, which told the tool what to omit.
`Optional` remains as a convention for secondary links.
## Why not /.well-known/
v2 explains why it does not use a `/.well-known/` address (RFC 8615). Such URIs exist only at the origin root. Many authors control only a path on a shared host and cannot add a file to the root. Like `index.html`, an `llms.txt` describes the path where it sits.
## What did not change
- Only the H1 is required.
- The order of parts: H1, blockquote, details without headings, H2 sections with link lists.
- The list item format: a link and optional notes after a colon.
- The file forbids nothing and does not control access.
## Absent from both revisions
- `llms-full.txt` and `llms-small.txt`. These files emerged as a practice of documentation platforms and are not in the proposal text.
- Requirements for `Content-Type`.
- A section on security and untrusted content.
- A rule for the language of the root file on a multilingual site.
The last three are open questions in the proposal repository.
## What a site owner should do
1. If the site already has an `llms.txt`, there is no need to redo it: the format is the same.
2. Check the links in the file: where possible they should point to Markdown versions of pages.
3. If Markdown versions exist, add `rel="alternate" type="text/markdown"` to the pages.
4. Add `rel="describedby"` pointing to the covering `llms.txt`.
5. Do not rely on any special handling of the `Optional` section.
:::note[What is known about adoption]
v2 is new. In a check of 46 sites on October 3, 2026, the `rel="describedby"` relation was found on one. Some documentation platforms send their own non-standard headers instead.
:::
## llms.txt example: basic files and templates
URL: https://llms-full-txt.ru/en/examples/basic/
Below are four sample `llms.txt` files for different sites. Their addresses and names are for teaching: the `.example` domain does not exist.
The structure is the same everywhere: a name, a summary, sections with links. What differs is what counts as the main material.
Copy the approach, not the text: which sections were chosen and what the notes say.
## The shortest valid file
Only the title is required:
```markdown
# Kolos
```
The file is formally valid and useless: an agent has nowhere to go. A working minimum is a name, a summary and at least one section.
```markdown
# Kolos
> A bakery in Leeds: bread and pastry with delivery.
## Main
- [Menu and prices](https://kolos.example/menu.md): the full range with ingredients
- [Delivery](https://kolos.example/delivery.md): areas, hours, cost
```
## A documentation site
- llms.txt
- Documentation of a service
- Name and summary
```
# Kolos API
> A delivery service for online stores: rate calculation, order creation, tracking.
```
- Who needs the service and why
- Know in advance
```
Every request needs a key in the Authorization header.
The API version is part of the address: /v2/.
```
- What an agent would get wrong in its very first request
- Where to start
```
## Start
- [Quick start](https://kolos.example/docs/start.md): the key, the first request, the first order
- [Authentication](https://kolos.example/docs/auth.md): keys, permissions, expiry
```
- The path of a newcomer
- Reference
```
## API
- [Orders](https://kolos.example/docs/orders.md): create, change, cancel
- [Rates](https://kolos.example/docs/rates.md): cost by address and weight
- [Errors](https://kolos.example/docs/errors.md): codes and what to do about them
```
- One link per subject
- Secondary
```
## Optional
- [Changelog](https://kolos.example/docs/changelog.md): versions and dates
```
- Can be skipped when a short context is needed
Why this way:
- Links point to Markdown versions of pages: an agent does not have to parse HTML.
- The "know in advance" block saves mistakes. It holds what applies to every request.
- The reference does not list every method: one page per subject.
## An online store
```markdown
# Volna
> An online store of home appliances: kettles, coffee makers, vacuum cleaners. Delivery across the UK.
Prices on the site include VAT. Availability is refreshed hourly.
## Catalog
- [Kettles](https://volna.example/category/kettles/): electric and stovetop, 1 to 2.5 l
- [Coffee makers](https://volna.example/category/coffee/): drip, espresso, capsule
- [Vacuum cleaners](https://volna.example/category/vacuum/): corded, stick, robots
## For customers
- [Delivery](https://volna.example/delivery/): cities, times, cost, pickup
- [Payment](https://volna.example/payment/): cards, instalments, pay on delivery
- [Returns and warranty](https://volna.example/returns/): terms, conditions, service address
- [Contacts](https://volna.example/contacts/): phone, address, opening hours
## Optional
- [About the company](https://volna.example/about/): legal details and history
```
Why this way:
- The file lists categories, not every product. There are thousands of products, and their list is already in `sitemap.xml`.
- The "For customers" section answers the most frequent questions: delivery, payment, returns.
- Prices are left out: they change. A machine takes the exact price and availability from the product page, from structured data.
What describes a product page: [Structured data and JSON-LD](https://llms-full-txt.ru/en/guides/json-ld/).
## A blog
```markdown
# Notes on the web
> A blog about how sites work: speed, accessibility, search. One author, a new post every two weeks.
## Main posts
- [Why a site is slow](https://notes.example/slow-site.md): five causes and how to find yours
- [Accessibility from zero](https://notes.example/a11y-start.md): what to check on any site in an hour
- [How to read a Lighthouse report](https://notes.example/lighthouse.md): which numbers matter and which do not
## Topics
- [Speed](https://notes.example/topics/speed/): posts about loading and rendering
- [Accessibility](https://notes.example/topics/a11y/): posts about keyboard, contrast, screen readers
## Optional
- [All posts](https://notes.example/archive/): the full list by date
- [About the author](https://notes.example/about/): experience and contacts
```
Why this way:
- The best posts come first, not the latest. RSS exists for the stream of new entries.
- The full archive is moved to `Optional`.
- The summary says who writes and how often: that helps judge the source.
## A company site
```markdown
# Opora Bureau
> We design and build timber-frame houses in Yorkshire. Fixed-price contracts.
## Services
- [Design](https://opora.example/design/): what a project includes, timing, what the client provides
- [Construction](https://opora.example/build/): stages, timing, warranty
- [Ready projects](https://opora.example/projects/): floor plans and areas
## How we work
- [Stages and payment](https://opora.example/process/): from survey to handover
- [Questions and answers](https://opora.example/faq/): plot, foundation, winter, mortgage
- [Contacts](https://opora.example/contacts/): address, phone, opening hours
```
Why this way:
- There is no section with news and vacancies: they do not answer a client's questions.
- The questions and answers page is a must: those are ready answers.
## What good files have in common
- The summary answers "what is here and for whom".
- There are few sections, named by meaning.
- Every link has a note.
- Secondary material sits in `Optional`.
- Nothing private and nothing extra.
## What to read next
- [Advanced examples](https://llms-full-txt.ru/en/examples/advanced/) - how the files of real sites are built
- [Best practices](https://llms-full-txt.ru/en/guides/best-practices/) - rules and typical mistakes
- [The file format](https://llms-full-txt.ru/en/specification/format/) - exact requirements
- [Getting started](https://llms-full-txt.ru/en/guides/getting-started/) - the path step by step
## llms.txt examples: advanced files explained
URL: https://llms-full-txt.ru/en/examples/advanced/
One file in the root suits a small site. When a project has several languages, versions, packages or services, there are several files, and the [second edition of the proposal](https://llmstxt.org/) says how an agent chooses between them: the most specific file that covers the page wins.
Below are six scenarios. Each has a diagram, a file structure, an example and a typical mistake.
| Scenario | How many files | Main idea |
|---|---|---|
| [Several languages](#several-languages) | one per language | a language file sits in the language subpath |
| [Several versions](#several-versions) | one per version | the root file leads to the current one |
| [Monorepo](#monorepo) | root and packages | the root is a table of contents, details live in packages |
| [Microservices](#microservices) | root and services | one entry, then by service |
| [Index and full text](#index-and-full-text) | two | the index is required, the full text is not |
| [Build in CI](#build-in-ci) | as many as needed | the file is rebuilt on every change |
## Several languages
- The site has several language versions in subpaths: /ru/, /es/.
- Putting all languages into one root file. An agent gets a mix and cannot tell which version it needs.
- The proposal does not define the language of the root file on a multilingual site: it is an open question (issue 147 in the repository).
- **llms.txt** the English version, covers the whole site
- ru/
- **llms.txt** the Russian version, covers /ru/
- es/
- **llms.txt** the Spanish version, covers /es/
**/llms.txt**
```markdown title="llms.txt"
# Documentation Hub
> Comprehensive guides and API reference for developers.
## Getting Started
- [Quick Start](https://docs.example.com/en/quickstart.md): 5-minute setup guide
- [Installation](https://docs.example.com/en/install.md): System requirements
## API Reference
- [REST API](https://docs.example.com/en/api/rest.md): HTTP endpoints
```
**/ru/llms.txt**
```markdown title="ru/llms.txt"
# Документация
> Подробные руководства и справочник API для разработчиков.
## Начало работы
- [Быстрый старт](https://docs.example.com/ru/quickstart.md): настройка за 5 минут
- [Установка](https://docs.example.com/ru/install.md): системные требования
## Справочник API
- [REST API](https://docs.example.com/ru/api/rest.md): HTTP-методы
```
## Several versions
- The documentation keeps several product versions: /docs/v3/, /docs/v2/.
- Leaving links to an old version in the root file. An agent will write code for an outdated API.
- What an agent reads
- A question without a version
- Root file
- /llms.txt
- Current version
- /docs/v3/
- A question about an old version
- Version file
- /docs/v2/llms.txt
- Pages of that version
- /docs/v2/
```markdown title="llms.txt"
# MyLibrary v3.0
> Modern JavaScript utility library (current stable).
## Documentation
- [Migration from v2](https://mylib.dev/docs/v3/migration.md): Breaking changes
- [API Reference](https://mylib.dev/docs/v3/api.md): Complete API docs
## Optional
- [v2 llms.txt](https://mylib.dev/docs/v2/llms.txt): Previous stable version
- [v1 llms.txt](https://mylib.dev/docs/v1/llms.txt): Deprecated
```
## Monorepo
- One site documents several packages, each with its own section.
- Listing every page of every package in the root. The file stops fitting into the context.
- OpenAI: a root file and nested indexes for /api, /codex and /commerce.
- **llms.txt** table of contents: one line per package
- core/
- **llms.txt** everything about the core
- cli/
- **llms.txt** everything about the command line
- plugins/
- **llms.txt** everything about plugins
```markdown title="llms.txt"
# Acme Platform
> Monorepo containing all Acme packages and services.
## Packages
- [Core](https://acme.dev/core/llms.txt): Core library and shared utilities
- [CLI](https://acme.dev/cli/llms.txt): Command-line interface
- [Plugins](https://acme.dev/plugins/llms.txt): Plugin system
## Guides
- [Getting Started](https://acme.dev/guides/quickstart.md): First steps
- [Architecture](https://acme.dev/guides/architecture.md): How the packages fit together
```
## Microservices
- The documentation describes a set of services with different APIs.
- Mixing public and internal services. Only what is open to everyone goes into the file: llms.txt does not restrict access.
- One entry, then by service
- Agent
- looks for how to accept a payment
- Root file
- /llms.txt
- the list of services
- Service file
- /billing/llms.txt
- billing pages
- The page
- /billing/payments.md
```markdown title="llms.txt"
# Platform API Documentation
> Microservices platform for e-commerce operations.
## Services
- [Auth](https://docs.platform.io/auth/llms.txt): Authentication and authorization
- [Users](https://docs.platform.io/users/llms.txt): User management
- [Orders](https://docs.platform.io/orders/llms.txt): Order processing
- [Billing](https://docs.platform.io/billing/llms.txt): Payments and invoices
## Optional
- [Status Page](https://status.platform.io): Service health
```
## Index and full text
- An agent sometimes needs the whole documentation text in one request.
- Treating llms-full.txt as part of the proposal. It is a practice of documentation platforms, and the file is not required.
- Stripe does without the full file. At Cloudflare it takes 67 MB.
- What each file holds
- llms.txt
- links
- section -> link -> link
- Read first
- Kilobytes
- llms-full.txt
- the page text itself
- page -> page -> page
- Read when the whole corpus is needed
- From hundreds of kilobytes to tens of megabytes
**llms.txt**
```markdown title="llms.txt"
# Framework Documentation
> Full-stack web framework for building modern applications.
## Core
- [Getting Started](https://framework.dev/docs/start.md): Installation and first app
- [Routing](https://framework.dev/docs/routing.md): File-based routing
- [Middleware](https://framework.dev/docs/middleware.md): Request and response middleware
## Optional
- [Full text](https://framework.dev/llms-full.txt): All pages in one file
```
**llms-full.txt**
```markdown title="llms-full.txt"
# Framework Documentation
> Full-stack web framework for building modern applications.
## Getting Started
Install the framework:
npm install framework
Create the first application...
## Routing
Routes are created from files in the pages directory...
## Middleware
Middleware functions have access to the request and the response...
```
## Build in CI
- The documentation changes often, and the file cannot be kept by hand.
- Rebuilding the file once a quarter. Links die sooner: three examples in the previous version of this page already answered 404.
- The file is built together with the site
- Change
- docs/**
- a page edit
- Build
- the generator reads the sitemap
- Check
- every link answers 200
- Publish
- public/llms.txt
```yaml title=".github/workflows/llms.yml" {16-19}
name: Update llms.txt
on:
push:
paths:
- 'docs/**'
schedule:
- cron: '0 0 * * 0'
jobs:
update:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Generate llms.txt
run: |
npx llmstxt gen https://docs.example.com/sitemap.xml \
-ep "**/blog/**" -t "My Docs" > public/llms.txt
- name: Commit and push
run: |
git config user.name "GitHub Action"
git config user.email "action@github.com"
git add public/llms.txt
git diff --staged --quiet || git commit -m "Update llms.txt"
git push
```
The generator commands are described on the [llmstxt CLI](https://llms-full-txt.ru/en/tools/llmstxt-cli/) page.
## Real files
Every address opened on the day of the check. For each one we say what the file shows and what it does not prove.
- [OpenAI](https://developers.openai.com/llms.txt): A root index plus nested files for /api, /codex, /commerce. Does not prove: that chatgpt reads other sites’ llms.txt.
- [Anthropic](https://platform.claude.com/llms.txt): A docs index and the full text in one file. Does not prove: that claude reads other sites’ llms.txt.
- [Cloudflare](https://developers.cloudflare.com/llms.txt): A very large index: the full text takes 67 MB. Does not prove: that a file this big is usable by an agent as a whole.
- [Stripe](https://docs.stripe.com/llms.txt): llms-full.txt is not required: an index and Markdown pages are enough. Does not prove: that skipping the full file is a general rule.
- [Cursor](https://cursor.com/llms.txt): The only measured site with rel=describedby from the second edition. Does not prove: that the second edition is widely adopted.
- [GitHub Docs](https://docs.github.com/llms.txt): An index without a full file on large docs. Does not prove: that copilot relies on this file.
- [Mintlify](https://www.mintlify.com/docs/llms.txt): The platform generates the files automatically for every site it hosts. Does not prove: the quality of each generated file.
- [Yandex Cloud](https://yandex.cloud/llms.txt): An example on a Russian site. Does not prove: that yandex search uses such files.
Checked October 3, 2026.
## What next
- [Basic examples](https://llms-full-txt.ru/en/examples/basic/) - where a small site starts.
- [File format](https://llms-full-txt.ru/en/specification/format/) - the rules of the second edition.
- [Best practices](https://llms-full-txt.ru/en/guides/best-practices/) - a checklist.
## llmstxt CLI: llms.txt and llms-full.txt from a sitemap
URL: https://llms-full-txt.ru/en/tools/llmstxt-cli/
[`llmstxt`](https://github.com/dotenvx/llmstxt) reads `sitemap.xml`, visits every page and assembles a list of links with titles and descriptions. The result is printed to the console and redirected into a file.
The utility suits a site that has a sitemap and no ready-made extension.
It does not run page scripts and does not decide what matters on the site. Selection stays with the owner.
- Author: dotenvx
- Package: llmstxt in the npm registry
- Version: 0.9.0, March 2026
- License: BSD-3-Clause
- Checked: October 3, 2026
## Running
No installation is needed:
```bash
npx -y llmstxt gen https://example.com/sitemap.xml > llms.txt
```
Always pass the sitemap address. Without it the utility falls back to a default address that is not yours.
## Commands
| Command | What it does |
|---|---|
| `gen` | builds `llms.txt`: a list of links with notes |
| `gen-full` | builds `llms-full.txt`: the full text of every page as Markdown |
```bash
npx -y llmstxt gen-full https://example.com/sitemap.xml > llms-full.txt
```
`gen-full` exists in the utility itself but is not mentioned in the project description. To see every command: `npx -y llmstxt --help`.
## Options
The options are the same for both commands.
| Option | Short | What it does |
|---|---|---|
| `--exclude-path` | `-ep` | exclude addresses by pattern |
| `--include-path` | `-ip` | keep only addresses matching a pattern |
| `--replace-title` | `-rt` | fix titles with a substitution rule |
| `--title` | `-t` | set the site name |
| `--description` | `-d` | set the site summary |
| `--concurrency` | `-c` | number of simultaneous requests, 5 by default |
## An example with selection
```bash
npx -y llmstxt gen https://example.com/sitemap.xml \
-ep "**/blog/**" \
-ep "**/privacy**" \
-ep "**/cart**" \
-rt 's/ \| My Site//' \
-t 'My Site' \
-d 'Documentation of a delivery service' \
> llms.txt
```
Here the blog, the policy and the cart are removed from the file, the site-name tail is cut from titles, and the name and summary are set by hand.
## What comes out
```markdown
# My Site
> Documentation of a delivery service
- [Quick start](https://example.com/docs/start/): the first request and the first order
- [API reference](https://example.com/docs/api/): methods, parameters, error codes
```
Notes are taken from page descriptions. When descriptions on the site are empty or identical, so are the notes.
## Limits
- **A sitemap is required.** The utility has no other input.
- **Scripts are not run.** Pages are fetched with a plain HTTP request. Content that appears only in a browser is invisible to the utility.
- **No sections.** The output is one list. Grouping by meaning is done by hand.
- **No selection.** Everything not excluded by patterns goes into the file.
- **Links point to HTML.** The utility does not create Markdown versions.
## Refreshing on a schedule
It is convenient to run the utility during the site build: the file is then refreshed together with the site.
```bash
npx -y llmstxt gen https://example.com/sitemap.xml -ep "**/cart**" > public/llms.txt
```
Run it after the new sitemap has been built.
## After generation
1. Open the file and read it: are the name and summary there.
2. Remove service pages with exclusion patterns.
3. Group the links into sections by hand.
4. Check the server response and the links.
More: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/), [best practices](https://llms-full-txt.ru/en/guides/best-practices/), [other tools](https://llms-full-txt.ru/en/tools/generators/).
## llmstxt-architect: generating llms.txt with an LLM
URL: https://llms-full-txt.ru/en/tools/llmstxt-architect/
`llmstxt-architect` crawls the pages of a site, hands the text of each to a language model and gets a note back: what the page is about and when it is worth reading. The notes are assembled into `llms.txt`.
Its strength is meaningful notes instead of copied page descriptions.
Its weakness is that [the project](https://github.com/rlancemartin/llmstxt_architect) has not been updated since March 2025. It needs access to a language model: a paid key or a local model.
no releases since March 2025
- Author: rlancemartin
- Package: llmstxt-architect on PyPI
- Version: 0.6.1, March 29, 2025
- License: MIT
- Checked: October 3, 2026
## Running
Without installation, through `uvx`:
```bash
uvx --from llmstxt-architect llmstxt-architect \
--urls https://example.com/docs \
--max-depth 1 \
--llm-name \
--llm-provider anthropic \
--project-dir out
```
Or installed into an environment:
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install llmstxt-architect
```
The model provider's key is passed as an environment variable, for example `ANTHROPIC_API_KEY` or `OPENAI_API_KEY`. A local model through Ollama additionally needs `langchain-ollama`.
The model name in the project's examples has aged together with the project. Put in a current model name from your provider.
## Options
| Option | Default | What it does |
|---|---|---|
| `--urls` | - | addresses to start crawling from |
| `--existing-llms-file` | - | take addresses from an existing `llms.txt`: a local path or an address |
| `--update-descriptions-only` | off | refresh only the notes, keeping structure and order |
| `--max-depth` | 5 | crawl depth |
| `--llm-name` | an old model | the model name |
| `--llm-provider` | `anthropic` | the model provider |
| `--project-dir` | `llms_txt` | the folder with results |
| `--output-dir` | `summaries` | the folder with per-page notes |
| `--output-file` | `llms.txt` | the name of the final file |
| `--summary-prompt` | built in | your own instruction for the model |
| `--blacklist-file` | none | a file with addresses to skip |
| `--extractor` | `default` | how text is extracted: `default` or `bs4` |
## Refreshing notes in an existing file
A useful mode: the structure is already thought through, and the notes are weak.
```bash
llmstxt-architect \
--existing-llms-file ./llms.txt \
--update-descriptions-only \
--llm-name \
--llm-provider anthropic \
--project-dir out
```
Without `--update-descriptions-only` the structure of the source file is not preserved.
## What comes out
```text
out/
llms.txt the final file
summaries/
.txt a note for each page
summarized_urls.json progress marks
```
A line of the final file looks like `[Title](address): a note from the model`. When a run is interrupted, repeating the same command resumes where it stopped.
The utility does not create `llms-full.txt`.
## Calling it from code
```python
import asyncio
from llmstxt_architect.main import generate_llms_txt
asyncio.run(generate_llms_txt(
urls=["https://example.com/docs"],
max_depth=1,
llm_name="",
llm_provider="anthropic",
project_dir="out",
))
```
## Limits
- **A language model is required.** Every page is a separate call, and on a large site that costs money.
- **No sections.** The output is one list of links.
- **Notes have to be read.** A model can describe a page inaccurately.
- **The project is not developing.** Nobody will fix bugs, and dependencies age.
## When it fits
The utility is justified for one-off work: assemble a draft of notes for large documentation, then fix it by hand. For continuous refreshing, take a tool that runs during the site build.
The same work can be done without the utility by handing a list of pages to an AI system: [a prompt for generation](https://llms-full-txt.ru/en/prompts/generate-llms-txt/).
Other options: [Generators and tools](https://llms-full-txt.ru/en/tools/generators/).
## llm.codes: documentation as clean Markdown for AI
URL: https://llms-full-txt.ru/en/tools/llm-codes/
`llm.codes` opens a documentation page in a real browser, waits for the scripts to finish and returns its content as clean Markdown.
It is not an `llms.txt` generator. The service solves a neighboring task: getting the text of documentation that looks empty without a browser.
It serves a reader of someone else's documentation, not a site owner.
- Author: amantus-ai
- Kind: an open-source web service
- License: MIT
- State: the site works, the code was updated in July 2026
- Checked: October 3, 2026
## Which task it solves
Some documentation sites assemble a page with scripts in the browser. A program that requests such a page with a plain HTTP request gets an empty shell.
- Two sides of one problem
- The site owner
- can fix the cause
- Serves text in the server-sent HTML
- Publishes Markdown versions of pages
- Publishes llms.txt
- The documentation reader
- works with what exists
- Cannot change someone else’s site
- Gets the text through a browser service
- llm.codes is one such service
If the site is yours, the right move is to serve content without scripts and add Markdown versions: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/).
## How to use it
1. Open [the service's site](https://llm.codes).
2. Paste the address of a documentation page.
3. Choose the crawl depth and the page limit.
4. Get the Markdown: copy or download it.
Settings in the interface:
| Setting | Values |
|---|---|
| Crawl depth | 0 to 5, 2 by default |
| Page limit | 1 to 2000, 200 by default |
| Code blocks only | drop the text, keep the examples |
| Filters | remove repeats and service lines |
Results stay in a cache for up to 30 days: a repeated request for the same page answers at once.
## Which sites it accepts
The service does not accept any address, only documentation addresses. The check uses patterns:
- documentation subdomains: `docs.*`, `developer.*`, `learn.*`;
- documentation paths: `/docs`, `/guide`, `/learn`;
- sites of languages and libraries;
- GitHub Pages.
On top of the patterns there is a short list of exceptions. An address outside the patterns is rejected.
## Self-hosting
[The code is open](https://github.com/amantus-ai/llm-codes), and the service can be run independently.
```bash
git clone https://github.com/amantus-ai/llm-codes.git
cd llm-codes
pnpm install
cp .env.local.example .env.local
pnpm run dev
```
It needs Node.js 24 and pnpm 10. Pages are loaded by one of two providers:
| Provider | What it needs |
|---|---|
| Firecrawl, the default | a paid `FIRECRAWL_API_KEY` |
| Playwright | no key; the browser is installed with `pnpm exec playwright install chromium` |
```ini
SCRAPE_PROVIDER=playwright
```
The Redis cache is optional.
## Limits
- **The service does not create `llms.txt`.** The output is page text, not a site map.
- **Not any site.** Only addresses that look like documentation are accepted.
- **There is no public API with keys and terms.** The internal endpoint is meant for the service's own interface.
- **Someone else's content stays theirs.** The rights to documentation text belong to its owner.
## What to read next
- [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/) - how a site owner can serve clean text directly
- [What is llms-full.txt](https://llms-full-txt.ru/en/guides/llms-full-txt/) - the full text of a site in one file
- [Generators and tools](https://llms-full-txt.ru/en/tools/generators/) - what to create `llms.txt` with
## llms.txt generators: tools and services
URL: https://llms-full-txt.ru/en/tools/generators/
A tool is chosen by what the site runs on. A content management system takes an extension, a documentation generator takes a plugin, and anything else takes a utility that reads `sitemap.xml`.
The best option is the one that assembles the file on publication: then it does not go stale.
The facts on this page were checked on October 3, 2026. Some tools from older lists have disappeared or been abandoned by that date.
## How to choose
| The site runs on | What to take | More |
|---|---|---|
| WordPress | a feature of an SEO plugin | [llms.txt in WordPress](https://llms-full-txt.ru/en/guides/wordpress/) |
| Webasyst, Shop-Script | an extension for the app | [llms.txt and JSON-LD in Webasyst](https://llms-full-txt.ru/en/guides/webasyst/) |
| Starlight, VitePress, Docusaurus, MkDocs, Nuxt | a documentation generator plugin | [Setup in documentation generators](https://llms-full-txt.ru/en/reference/configuration/) |
| [Mintlify](https://www.mintlify.com/docs/ai/llmstxt), [GitBook](https://gitbook.com/docs/getting-started/llm-ready-docs) | nothing: the platform creates the files | - |
| Any site with `sitemap.xml` | a command-line utility | [llmstxt CLI](https://llms-full-txt.ru/en/tools/llmstxt-cli/) |
| A site with no system and no sitemap | write it by hand | [Getting started](https://llms-full-txt.ru/en/guides/getting-started/) |
## Content management systems
| System | Tool | What it creates | State |
|---|---|---|---|
| WordPress | [Yoast SEO](https://yoast.com/features/llms-txt/) | `llms.txt` | maintained |
| WordPress | [All in One SEO](https://aioseo.com/features/llms-txt/) | `llms.txt`, `llms-full.txt` | maintained |
| WordPress | [Rank Math](https://rankmath.com/kb/llms-txt/) | `llms.txt`, a module enabled by hand | maintained |
| WordPress | [Website LLMs.txt](https://wordpress.org/plugins/website-llms-txt/) | `llms.txt`, optional Markdown versions | maintained, over 40 thousand installs |
| Webasyst | extensions for six apps | `llms.txt`, `llms-full.txt`, section files | maintained |
| Drupal | the `llmstxt` module | `llms.txt` | project page available |
| Wix | a built-in feature | `llms.txt` | enabled by the platform |
## Documentation generators
| Generator | Package | `llms-full.txt` | Markdown versions of pages |
|---|---|---|---|
| Starlight | `starlight-llms-txt` | yes, plus `llms-small.txt` | not stated |
| VitePress | `vitepress-plugin-llms` | yes | yes |
| Docusaurus | `docusaurus-plugin-llms` | yes | by option |
| Docusaurus | `@signalwire/docusaurus-plugin-llms-txt` | by option | yes |
| MkDocs | `mkdocs-llmstxt` | by option | for listed files |
| Nuxt | `nuxt-llms` | by option | no |
A working setup for each: [Setup in documentation generators](https://llms-full-txt.ru/en/reference/configuration/).
The author of `mkdocs-llmstxt` has declared the project to be in maintenance mode: no new features.
## Command-line utilities
| Utility | Language | How it works | State |
|---|---|---|---|
| [llmstxt](https://llms-full-txt.ru/en/tools/llmstxt-cli/) | Node.js | reads `sitemap.xml`, takes page titles and descriptions | release 0.9.0, March 2026 |
| [llmstxt-architect](https://llms-full-txt.ru/en/tools/llmstxt-architect/) | Python | crawls a site and writes link notes with a language model | no releases since March 2025 |
| `llms-txt` | Python | the library by the author of the proposal: parsing the file | release 0.0.7, September 2026 |
| Crawl4AI | Python | a crawler that returns pages as Markdown; it does not build `llms.txt` itself | maintained |
## Web services
| Service | What it does | State |
|---|---|---|
| [llm.codes](https://llms-full-txt.ru/en/tools/llm-codes/) | turns documentation pages into Markdown; does not create `llms.txt` | working |
| the Firecrawl generator | built `llms.txt` and `llms-full.txt` from a site address | repository unchanged since June 2025 |
| llmstxt.com | a generator and a directory | the site answers |
removed from the list
Tools that did not exist on the check date were removed from the earlier version of this page: two WordPress generators, the Jeeves plugin, the `llmstxt-parser` package, the `llms-txt-php` library at its former address. The `mcpdoc` server was archived by its owner: [llms.txt and MCP](https://llms-full-txt.ru/en/reference/mcp-relationship/).
## Directories of sites with llms.txt
Directories collect the addresses of published files. The largest open one is `llms-txt-hub`. There are also `llmstxthub.com` and `directory.llmstxt.cloud`.
Use them with a caveat: entries age, and some listed sites no longer serve the file. A checked list: [Who publishes llms.txt](https://llms-full-txt.ru/en/reference/adoption/).
## What to check after any tool
1. The file answers `200` and is served as text.
2. It has the site name and summary, not only a list.
3. No service or private pages.
4. Links have notes, and the notes make sense.
5. It is clear when the file will be rebuilt next.
A generator does not know what matters on the site. Read an automatically assembled file and fix the selection: [best practices](https://llms-full-txt.ru/en/guides/best-practices/), [validation](https://llms-full-txt.ru/en/guides/validation/).
## llms.txt in Starlight, VitePress, Docusaurus, MkDocs, Nuxt
URL: https://llms-full-txt.ru/en/reference/configuration/
In documentation generators a plugin creates `llms.txt` during the site build. The setup takes a few lines, and the file is refreshed with every publication.
Below is one working setup for each of five generators. Package names and versions were checked on October 3, 2026.
On the [Mintlify](https://www.mintlify.com/docs/ai/llmstxt) and [GitBook](https://gitbook.com/docs/getting-started/llm-ready-docs) platforms there is nothing to set up: the files are created automatically.
## What each plugin creates
| Generator | Package | `llms.txt` | `llms-full.txt` | Markdown versions of pages |
|---|---|---|---|---|
| Starlight | `starlight-llms-txt` | yes | yes, plus `llms-small.txt` | not stated |
| VitePress | `vitepress-plugin-llms` | yes | yes | yes |
| Docusaurus | `docusaurus-plugin-llms` | yes | yes | by option |
| Docusaurus | `@signalwire/docusaurus-plugin-llms-txt` | yes | by option | yes |
| MkDocs | `mkdocs-llmstxt` | yes | by option | for listed files |
| Nuxt | `nuxt-llms` | yes | by option | no |
## Setup
**Starlight**
```bash
pnpm add starlight-llms-txt
```
```js
// astro.config.mjs
import starlight from '@astrojs/starlight'
import { defineConfig } from 'astro/config'
import starlightLlmsTxt from 'starlight-llms-txt'
export default defineConfig({
site: 'https://example.com/',
integrations: [
starlight({
title: 'My Docs',
plugins: [starlightLlmsTxt()],
}),
],
})
```
The `site` field is required: without it the plugin cannot build absolute addresses. The output is three files: `llms.txt`, `llms-full.txt` and a shortened `llms-small.txt`.
**VitePress**
```bash
npm install vitepress-plugin-llms --save-dev
```
```ts
// .vitepress/config.ts
import { defineConfig } from 'vitepress'
import llmstxt from 'vitepress-plugin-llms'
export default defineConfig({
vite: {
plugins: [llmstxt()],
},
})
```
The plugin creates `llms.txt`, `llms-full.txt` and a Markdown version of every page.
The plugin's author advises using it for English documentation only. For a multilingual site, take that limit into account in advance.
**Docusaurus**
There are two live plugins. The first is simpler to set up:
```bash
npm install docusaurus-plugin-llms --save-dev
```
```js
// docusaurus.config.js
module.exports = {
plugins: ['docusaurus-plugin-llms'],
}
```
The second can do more and creates Markdown versions of pages:
```bash
npm install @signalwire/docusaurus-plugin-llms-txt
```
```js
// docusaurus.config.js
module.exports = {
plugins: [
[
'@signalwire/docusaurus-plugin-llms-txt',
{
outputDir: 'llms-txt',
includePatterns: ['**/*.html'],
excludePatterns: ['**/404.html'],
},
],
],
}
```
The similarly named unscoped package `docusaurus-plugin-llms-txt` has not been updated since December 2024. It is not worth taking.
**MkDocs**
```bash
pip install mkdocs-llmstxt
```
```yaml
# mkdocs.yml
site_name: My project
site_description: Description of my project.
site_url: https://example.com/
plugins:
- llmstxt:
markdown_description: Long description of my project.
full_output: llms-full.txt
sections:
Documentation:
- index.md: Where to start
- api.md: API reference
```
The `site_url` field is required. Sections and files are listed by hand: this is the only plugin of the five that forces a selection. The `full_output` line enables `llms-full.txt`.
The author has declared the project to be in maintenance mode.
**Nuxt**
```bash
npm i nuxt-llms
```
```ts
// nuxt.config.ts
export default defineNuxtConfig({
modules: ['nuxt-llms'],
llms: {
domain: 'https://example.com',
title: 'My Site',
description: 'What the site is and who it is for.',
full: {
title: 'Full documentation',
description: 'The whole documentation text in one file.',
},
},
})
```
Only the `domain` field is required. The `full` block enables `llms-full.txt`.
## Platforms with nothing to set up
| Platform | What is created automatically |
|---|---|
| Mintlify | `llms.txt` and `llms-full.txt` at the project root; a very long index is split into parts |
| GitBook | `llms.txt`, `llms-full.txt` and a Markdown version of any page at the address ending in `.md` |
This explains why so many documentation sites have the files although their owners configured nothing.
## Discovery relations
Plugins create the files, but not all of them add the relations from the [second edition of the proposal](https://llmstxt.org/) to pages. Check whether the page code has these lines:
```html
```
If not, add them through the page head template. More: [Discovery of the file and Markdown versions](https://llms-full-txt.ru/en/specification/discovery/).
## After the build
1. `llms.txt` appeared in the build folder and answers `200` on the site.
2. It has the site name and summary.
3. Link addresses are absolute and on the right domain.
4. Drafts and hidden pages did not get into the file.
5. The size of `llms-full.txt` is reasonable.
6. Markdown versions of pages open, if the plugin creates them.
Commands for checking: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/).
## What to read next
- [Generators and tools](https://llms-full-txt.ru/en/tools/generators/) - extensions for content management systems and utilities
- [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/) - why they exist and how to serve them
- [What is llms-full.txt](https://llms-full-txt.ru/en/guides/llms-full-txt/) - when the full file is needed
- [llms.txt and MCP](https://llms-full-txt.ru/en/reference/mcp-relationship/) - how the file relates to the protocol
## llms.txt vs MCP: the difference and how they relate
URL: https://llms-full-txt.ru/en/reference/mcp-relationship/
`llms.txt` and MCP are two different systems. `llms.txt` is a file: an agent downloads and reads it. MCP is a protocol: a client program talks to a server, calls tools and receives data.
They do not depend on each other. A separate bridge can connect them, but that is a property of neither the file nor the protocol.
- Two independent systems
- llms.txt
- what is on the site
- Site documentation -> llms.txt
- A static Markdown file
- Any HTTP client will do
- Read only
- Made in 15 minutes
- MCP
- give me data and perform an action
- MCP client -> MCP server
- A protocol on top of JSON-RPC 2.0
- Needs a client and a server
- Read, write and tools
- Needs server development
- An optional bridge
- A separate MCP server that the user gives a list of llms.txt files. An example is mcpdoc.
- llms.txt -> bridge server -> assistant in an editor
## What MCP is
[Model Context Protocol](https://modelcontextprotocol.io/) is an open protocol that connects language models to external data and tools. Anthropic created it in November 2024. In December 2025 the protocol was donated to the Agentic AI Foundation under the Linux Foundation.
Through MCP a model can read files and databases, call APIs, run tools and get context from external sources.
## How they differ
| | llms.txt | MCP |
|---|---|---|
| What it is | a file | a protocol |
| Format | Markdown | JSON-RPC 2.0 |
| How it is obtained | one GET request | a session between a client and a server |
| What is needed | any HTTP client | an MCP client and server |
| What is possible | reading | reading, writing, calling tools |
| Risks | as with any public page | tool poisoning, prompt injection |
## The bridge between them
- How llms.txt reaches an assistant through MCP
- Site
- publishes llms.txt
- llms.txt
- svelte.dev/llms.txt
- Bridge server
- mcpdoc
- the user provides the list of files
- Assistant
- gets pages as a tool
[mcpdoc](https://github.com/langchain-ai/mcpdoc) (langchain-ai) is a separate MCP server. The repository is archived and no longer developed; the example below shows the principle.
1. **The site publishes llms.txt**
```markdown title="svelte.dev/llms.txt"
# Svelte Documentation
> Svelte is a UI framework that compiles components into efficient JavaScript.
## Docs
- [Overview](https://svelte.dev/docs/svelte/overview): What Svelte is and how it works
- [SvelteKit introduction](https://svelte.dev/docs/kit/introduction): The application framework
```
2. **The user connects the bridge in the assistant settings**
```json title="mcp.json"
{
"mcpServers": {
"docs": {
"command": "uvx",
"args": [
"--from", "mcpdoc",
"mcpdoc",
"--urls",
"Svelte:https://svelte.dev/llms.txt",
"Stripe:https://docs.stripe.com/llms.txt",
"--transport", "stdio"
]
}
}
}
```
3. **The assistant requests documentation**
It gets the list of links from `llms.txt`, loads the pages it needs and answers from fresh documentation.
Another tool of the same kind is [mcp-llms-txt-explorer](https://github.com/thedaviddias/mcp-llms-txt-explorer): an MCP server for navigating the `llms.txt` of any site.
## Security
MCP introduces threats a static file does not have.
| Threat | MCP | llms.txt |
|---|---|---|
| Tool poisoning | a malicious server injects dangerous instructions | not applicable: there are no tools |
| Prompt injection | server data changes model behavior | possible, as on any page |
| Behavior change after trust | the server changes after it is connected | the file can change but performs no actions |
| Data exfiltration | through tool calls | not applicable |
**Confirmed**
- In the [Common Crawl](https://commoncrawl.org/blog) corpus of 584 thousand files, 10 contain a classic prompt injection and 3,793 softly steer the model.
- `llms.txt` performs no actions: it is text.
**Not proven**
- That the file is safe by itself. The proposal text has no security section; the question is discussed in the repository.
**What it does not mean**
- "Read only" does not mean "trusted". An agent must treat the file content as untrusted, like any text from the network.
## What to choose
- An agent has to find documentation and read it: `llms.txt` is enough.
- A model has to perform actions or get private data: MCP is needed.
- Documentation is already described in `llms.txt`, and the assistant works only through MCP: a bridge fits.
## Live examples
Sites that have both `llms.txt` and material about MCP:
- [mcpdoc.ru](https://mcpdoc.ru) - documentation on MCP servers ([llms.txt](https://mcpdoc.ru/llms.txt))
- [n8n-mcp.ru](https://n8n-mcp.ru) - n8n with MCP ([llms.txt](https://n8n-mcp.ru/llms.txt))
## Who publishes llms.txt: a registry of sites
URL: https://llms-full-txt.ru/en/reference/adoption/
`llms.txt` is published first of all by developer documentation sites: OpenAI, Anthropic, Stripe, Cloudflare, GitHub. The bulk of files is created not by people but by platforms and plugins that switch the feature on for all their sites at once.
Adoption estimates range from 2% to 28%: they depend on which sites made it into the sample.
Publishing your own file does not mean a company reads other sites' files. No owner of an AI system has stated such a commitment.
## How many sites have the file
- 2.1%
- of sites in the HTTP Archive crawl: 8.7%
- of the top thousand sites: 28%
- of domains in a sample skewed towards an SEO audience: 584K
- files found in a single crawl of the web
The numbers do not contradict one another: they have different denominators. A crawl of the whole web gives about two percent. A sample of sites whose owners do SEO gives ten times more, and its authors call it an upper bound themselves.
Notes on the sources: [Web Almanac](https://llms-full-txt.ru/en/blog/web-almanac-2025-llms-txt/), [Common Crawl](https://llms-full-txt.ru/en/blog/common-crawl-584k-llms-txt/).
## Who creates the files
| Source of the file | Share | Where the number comes from |
|---|---|---|
| Made by a generator, not by hand | 68% | [Common Crawl](https://commoncrawl.org/blog) |
| Sites on Wix | 41% | Common Crawl |
| All in One SEO for WordPress | 39.6% | [Web Almanac 2025](https://almanac.httparchive.org/en/2025/seo) |
| Yoast SEO for WordPress | 3.6% | Web Almanac 2025 |
The shares come from different crawls and do not add up. The common conclusion: most files exist because a platform switched the feature on by default.
## Sites that were checked
The measurement was made by requests to the sites themselves. The table notes what each example shows and what it does not prove.
- [OpenAI](https://developers.openai.com/llms.txt): A root index plus nested files for /api, /codex, /commerce. Does not prove: that chatgpt reads other sites’ llms.txt.
- [Anthropic](https://platform.claude.com/llms.txt): A docs index and the full text in one file. Does not prove: that claude reads other sites’ llms.txt.
- [Cloudflare](https://developers.cloudflare.com/llms.txt): A very large index: the full text takes 67 MB. Does not prove: that a file this big is usable by an agent as a whole.
- [Stripe](https://docs.stripe.com/llms.txt): llms-full.txt is not required: an index and Markdown pages are enough. Does not prove: that skipping the full file is a general rule.
- [Cursor](https://cursor.com/llms.txt): The only measured site with rel=describedby from the second edition. Does not prove: that the second edition is widely adopted.
- [GitHub Docs](https://docs.github.com/llms.txt): An index without a full file on large docs. Does not prove: that copilot relies on this file.
- [Mintlify](https://www.mintlify.com/docs/llms.txt): The platform generates the files automatically for every site it hosts. Does not prove: the quality of each generated file.
- [Yandex Cloud](https://yandex.cloud/llms.txt): An example on a Russian site. Does not prove: that yandex search uses such files.
Checked October 3, 2026.
## Who does not have the file
On the measurement date `/llms.txt` answered `404` at:
| Site | Why it is notable |
|---|---|
| developers.google.com, developer.chrome.com, web.dev | Lighthouse checks for the file, but the team's own sites do not publish it |
| docs.astro.build, starlight.astro.build | often cited as examples in older lists |
| developer.mozilla.org, the root of huggingface.co | the same |
| tailwindcss.com | developer documentation without the file |
| ahrefs.com, moz.com | they study the format but do not publish it |
The list is a useful correction to "who supports `llms.txt`" directories: they age fast and are rarely rechecked.
## The second edition
The [second edition of the proposal](https://llmstxt.org/) came out on August 10, 2026 and added discovery relations: `rel="describedby"` and `rel="alternate"`. Among the measured sites `rel="describedby"` appeared on one. Documentation platforms still send headers of their own that are not in the proposal.
What changed: [changes in the second edition](https://llms-full-txt.ru/en/specification/changes/), [discovery](https://llms-full-txt.ru/en/specification/discovery/).
## What follows from this
**Confirmed**
- Large documentation sites publish `llms.txt` and Markdown versions of pages.
- Most files on the web were created automatically.
- Not everyone publishes `llms-full.txt`: Stripe and GitHub Docs do without it.
**Not proven**
- That owners of AI systems read other sites' files: nobody has published such a commitment.
- That the second edition is widespread: the measurement shows isolated cases.
**What it does not mean**
- A well-known company having the file does not prove a benefit for another site.
- The number of published files is not the number of files read.
## How the measurement was made
- Date: {CHECKED.en}.
- Method: requests to `/llms.txt` and `/llms-full.txt`, a page request with `Accept: text/markdown`, inspection of the `Link` header and `rel` links in the HTML.
- The status code, content type and size were recorded.
- The answer depends on the client: some sites answer a command line and redirect a browser. The table records the answer given to a command line.
Repeat the measurement for your own site: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/).
## What to read next
- [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/) - studies of requests to the files
- [Advanced examples](https://llms-full-txt.ru/en/examples/advanced/) - how real files are built
- [Status of the standard](https://llms-full-txt.ru/en/guides/standard-status/) - what a proposal is and who stands behind it
## Glossary: llms.txt, GEO, AEO and AI crawler terms
URL: https://llms-full-txt.ru/en/reference/glossary/
This subject has many words, and they are not equal. Some are fixed in documents and code, some were coined by service vendors, some are product names.
Each term below carries a status: where the word came from and whether it can be relied on.
document or standard · proposal · convention · research paper · vendor word
## Files and formats
### llms.txt
proposal
A text file at the root of a site: a name, a short summary and lists of links to the main material. Written for an AI agent. Proposed by Jeremy Howard in September 2024; the [second edition](https://llmstxt.org/) came out on August 10, 2026. It is not an IETF or W3C standard.
More: [What is llms.txt](https://llms-full-txt.ru/en/guides/what-is-llms-txt/), [the format](https://llms-full-txt.ru/en/specification/format/).
### llms-full.txt
convention
A file with the full text of a site's material in one document. It is not in the `llms.txt` proposal: documentation platforms established the name.
More: [What is llms-full.txt](https://llms-full-txt.ru/en/guides/llms-full-txt/).
### robots.txt
standard
A file with access rules for robots, described in RFC 9309. It says where robots may go, not what is useful.
More: [AI crawlers and robots.txt](https://llms-full-txt.ru/en/guides/ai-crawlers/).
### sitemap.xml
standard
A list of site addresses for search robots. It helps discover pages and does not rate their importance.
### Markdown
convention
A simple text format: headings, lists, links and tables are marked with ordinary characters. Convenient for a person and for a program.
More: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/).
### Structured data
search engine documentation
A machine-readable description of a page: what is on it and which properties it has.
More: [Structured data and JSON-LD](https://llms-full-txt.ru/en/guides/json-ld/).
### Schema.org
shared vocabulary
A vocabulary of types and properties for structured data: `Product`, `Article`, `Organization`, `price`, `author`. Founded in 2011 by the search engines Google, Bing, Yahoo and Yandex.
### JSON-LD
W3C recommendation
A format for writing structured data: a separate block in the page code. Google recommends it. The other formats are Microdata and RDFa.
## Programs
### Crawler
owner documentation
A program that walks sites and downloads pages. AI companies run crawlers of three classes: for model training, for a search index, and for a specific user's request.
More: [AI crawlers and robots.txt](https://llms-full-txt.ru/en/guides/ai-crawlers/).
### AI agent
convention
A program built on a language model that searches, reads pages and performs actions on a person's request. It differs from a crawler in that it comes for the answer to a specific question.
### MCP
open protocol
Model Context Protocol - a protocol through which an AI application connects to external data and tools. It is not directly related to `llms.txt`.
More: [llms.txt and MCP](https://llms-full-txt.ru/en/reference/mcp-relationship/).
### Context
convention
The amount of text a model can take into account at once. It is limited, which is why short maps and clean text without service markup suit an agent.
## Optimization and measurement
### GEO
research paper
Generative Engine Optimization - optimization for generative search systems. The term was introduced in a [2023 research paper](https://arxiv.org/abs/2311.09735). What is measured is the share and position of a source in an answer.
More: [GEO and AEO](https://llms-full-txt.ru/en/guides/geo-aeo/).
### AEO
vendor word
Answer Engine Optimization - optimization for systems that answer rather than return a list of links. Its origin is attributed to 2017; the primary source is not confirmed.
### LLMO
vendor word
Large Language Model Optimization. Used as a synonym of GEO. No primary source was found.
### AI SEO
vendor word
An ambiguous term. The first meaning is using AI in SEO work. The second is working on a site's presence in AI answers. Without clarification it is unclear which one is meant.
### AI visibility
vendor word
An umbrella name for a site's presence in AI answers. It is not measured as such: bot requests, referrals, citations and share of voice are.
More: [How to measure AI visibility](https://llms-full-txt.ru/en/guides/ai-visibility/).
### Share of voice
convention
The share of answers that name a site or a brand over a fixed set of questions. A comparative metric.
### Citation
convention
A link to a site in an AI answer as a source of facts.
## Products and names
### AI Overviews
product name
A block with an answer above the results in Google Search. [Introduced on May 14, 2024](https://blog.google/products/search/generative-ai-google-search-may-2024/). Search Console has no separate breakdown for it.
### AI Mode
product name
A mode of Google Search with a dialog instead of a list of links. [Introduced on March 5, 2025](https://blog.google/products/search/ai-mode-search/).
### Agentic web
vendor word
The idea of a web where agents, not people, perform a large share of actions. [Microsoft brought the word into use in May 2025](https://blogs.microsoft.com/blog/2025/05/19/microsoft-build-2025-the-age-of-ai-agents-and-building-the-open-agentic-web/).
### Agentic Browsing
category name
An experimental category of audits in Lighthouse that includes the `llms.txt` audit.
More: [Lighthouse and llms.txt](https://llms-full-txt.ru/en/guides/lighthouse/).
## What Google calls all this
In its guide [Google writes](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) that AEO and GEO are terms you may see used, while optimizing for AI features of Search is still SEO. The company does not treat it as a separate discipline.
## Paying for Claude and ChatGPT from Russia
URL: https://llms-full-txt.ru/en/reference/from-russia/
Almost every step in this reference runs into a foreign service: checking an AI answer, building a file in an editor, publishing the site, looking at mentions.
There are two obstacles. The service does not accept a Russian card, and some services do not open from Russia. Below is what closes both, by task.
Categories: [AI models](#ai-models) · [API keys](#api-keys) · [Code editors and agents](#code-editors-and-agents) · [Site builders](#site-builders) · [Hosting and builds](#hosting-and-builds) · [Clouds](#clouds) · [Domains](#domains) · [SEO services](#seo-services) · [Automation](#automation) · [Team](#team) · [Credentials](#credentials) · [Text and images](#text-and-images) · [When a service does not open](#when-a-service-does-not-open) · [What is available in Russia](#what-is-available-in-russia)
## How payment works
The payment links lead to a Russian-language service. The account is topped up in rubles, and the subscription or balance of the foreign service is paid from it. Each service has its own page with the steps.
A subscription and an API are separate accounts. A ChatGPT subscription does not pay for an OpenAI API key, and the reverse: if a utility needs a key, pay for the API.
## AI models
They are needed to check what they answer about your site and to run the [prompts](https://llms-full-txt.ru/en/prompts/). How to count mentions: [How to measure AI visibility](https://llms-full-txt.ru/en/guides/ai-visibility/).
## API keys
A key is needed by utilities such as [llmstxt-architect](https://llms-full-txt.ru/en/tools/llmstxt-architect/) and by your own scripts. It is a separate account, not a chat subscription.
## Code editors and agents
An editor with a model inside is a handy place to build and check `llms.txt` right in the site repository.
## Site builders
For when there is no site yet. What a finished site should have: [An AI-ready site](https://llms-full-txt.ru/en/guides/ai-ready-site/).
## Hosting and builds
The file does not go stale when it is built together with the site. On Markdown for agents: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/).
## Clouds
Storage, servers and models by API from the large providers.
## Domains
`llms.txt` is read from the domain root, so your own domain matters more than an address on someone else's platform.
## SEO services
They show rankings, links and mentions of a site. What in their numbers can be trusted: [AI Search Optimization](https://llms-full-txt.ru/en/guides/geo-aeo/).
## Automation
Rebuild the file on a schedule, check links, send a report - with no code of your own.
## Team
Where tasks, designs and the knowledge base live.
## Credentials
API keys do not belong in a repository - a password manager is the place for them.
## Text and images
Tools for site materials.
## When a service does not open
A paid subscription does not help when the service is closed to Russian addresses. Access is set up first, payment comes second.
## What is available in Russia
Alice answers can be checked with no payment and no network setup. A site on Webasyst has [extensions for llms.txt and JSON-LD](https://llms-full-txt.ru/en/guides/webasyst/).
## Yandex search in any browser
Whether a site made it into Alice's answers and Yandex results is easiest to check straight from the address bar. Yandex is not built in everywhere, though: Firefox does not list it among its search engines on desktop or on phones. There are two ways to bring it back.
**With an add-on.** Firefox on desktop and on Android has the [Yandex Search](https://addons.mozilla.org/ru/firefox/addon/yandexsearchengineextension/) add-on. It makes Yandex the default search in the address bar and adds the `@ya` keyword for a query from any tab.
**By hand.** This works in any browser, including those that take no add-ons, such as Firefox on iPhone:
1. Open the browser settings and the Search section.
2. Choose "Add search engine".
3. Type "Yandex" as the name and paste the search address into the URL field:
```text
https://ya.ru/search/?clid=15437607&text=%s
```
4. Save and pick Yandex as the default search engine.
The same steps in Firefox on iPhone:
In Chrome and browsers built on it the same settings section is called "Search engine", and a new entry is added under "Manage search engines".
## Generate llms.txt
URL: https://llms-full-txt.ru/en/prompts/generate-llms-txt/
Input: Name, description and a list of site pages
Output: A ready llms.txt file
## Why
Get a first draft of `llms.txt` in a minute. The model sorts links into sections and writes the notes; you check the addresses.
## When to use
- The site has no file yet, and the list of important pages is clear.
- You need to show colleagues what the file will look like.
For a large site a [generator](https://llms-full-txt.ru/en/tools/generators/) is more convenient: it collects pages from the sitemap.
## What to fill in
| Variable | What it is | Example |
|---|---|---|
| `[name]` | project or site name | Acme Docs |
| `[description]` | one or two sentences about the project | Payment API documentation |
| `[pages]` | addresses and titles of the main pages | a list of 10-30 lines |
## Prompt
```text title="Prompt"
You know the llms.txt format by the second edition of the proposal (llmstxt.org).
Write an llms.txt file for the project.
Name: [name]
Description: [description]
Pages:
[pages]
Rules:
1. The first line is an H1 heading with the project name.
2. Then a blockquote with a one or two sentence description.
3. Then sections with H2 headings, each with a list of links.
4. Link format: - [Title](address): a short note.
5. Put secondary links into the Optional section.
6. Use only addresses from my list. Do not invent anything.
Return only the file content, no explanations.
```
## What you get
```markdown title="llms.txt"
# Acme Docs
> Documentation of the Acme payment API.
## Documentation
- [Quick start](https://acme.dev/quickstart.md): the first payment in five minutes
- [API reference](https://acme.dev/api.md): methods and parameters
## Optional
- [Changelog](https://acme.dev/changelog.md)
```
## Check the result
1. Every address comes from your list and opens.
2. There is one H1 heading.
3. There are no headings between the description and the first section.
4. The file has no promises or advertising phrases.
## Audit llms.txt
URL: https://llms-full-txt.ru/en/prompts/audit-llms-txt/
Input: The content of your llms.txt
Output: A list of problems and a fixed file
## Why
Find format violations and weak spots in a finished file before an agent stumbles on them.
## When to use
- Before publishing the file.
- After a large site update.
- When a generator built the file and nobody has read it.
The model will not open the links by itself. Check that the addresses are alive separately.
## What to fill in
| Variable | What it is |
|---|---|
| `[file]` | the full content of `llms.txt` |
## Prompt
```text title="Prompt"
Check an llms.txt file against the second edition of the proposal (llmstxt.org).
[file]
Check in order:
1. The first heading is H1, and there is only one.
2. It is followed by a blockquote with a description.
3. There are no headings between the description and the first H2 section.
4. Every section is an H2 heading and a list of links.
5. Links have clear titles and notes.
6. Secondary links are in Optional.
7. The file has no instructions for the model and no advertising phrases.
Return:
- a table "problem - line - how to fix";
- a corrected version of the file.
Do not change link addresses and do not add new ones.
```
## What you get
A table of remarks and a corrected file. Do not ask for a score out of ten: a number without criteria means nothing.
## Check the result
1. The model did not change link addresses.
2. Every remark points at a specific line.
3. The corrected file starts with H1.
## Optimize content for AI
URL: https://llms-full-txt.ru/en/prompts/optimize-for-ai/
Input: The text of one page
Output: A rewritten page and a list of edits
## Why
Make a page clearer for a model: a plain answer at the top, a clear structure, facts with sources. It does not guarantee citations in AI answers - nobody can promise that.
## When to use
- The page answers a specific question, but the answer is buried in the middle.
- The text has a lot of filler and few facts.
- Headings do not say what is under them.
## What to fill in
| Variable | What it is |
|---|---|
| `[question]` | the question the page answers |
| `[text]` | the full text of the page |
## Prompt
```text title="Prompt"
Rewrite the page so that both a person and a language model can understand it easily.
The question the page answers: [question]
Text:
[text]
Rules:
1. In the first paragraph give a direct answer in two or three sentences.
2. Every heading must say what is written under it.
3. Give definitions in one phrase: "X is Y".
4. Use lists for enumerations and tables for comparisons.
5. Remove filler and repetition.
6. Do not change or add facts and numbers. If a claim has no source, mark it.
Return:
- the rewritten text in Markdown;
- a list of edits: what changed and why;
- a list of claims without a source.
```
## What you get
A page with the answer in the first paragraph, a list of edits and a list of places that need a source.
## Check the result
1. The model did not add facts or numbers that were not there.
2. The meaning of the text did not change.
3. Claims without a source are either confirmed or removed.
## llms.txt in autumn 2026: what is known
URL: https://llms-full-txt.ru/en/blog/state-of-llms-txt-autumn-2026/
Published: 2026-10-03
In nine months of 2026 more became known about `llms.txt` than in the year and a half before. Measurements replaced opinions. Below is a summary: what happened and where it led.
## What happened
| Date | Event |
|---|---|
| January 15 | [Web Almanac](https://almanac.httparchive.org/en/2025/seo): 2.1% of sites have the file |
| February 12 | Cloudflare serves Markdown to agents by the `Accept` header |
| May 7 | Lighthouse checks `llms.txt` by default |
| June 15 | Google: Search does not use such files |
| June 15 | [Ahrefs](https://ahrefs.com/blog/llmstxt-study): 97% of the files were never requested |
| August 10 | the second edition of the proposal |
| August 31 | [Common Crawl](https://commoncrawl.org/blog): 23% of the files have no links |
| September 16 | Mintlify: agents overtook people in documentation |
## What is confirmed
- Large documentation sites publish the file, including the developers of AI systems.
- Agents in code editors work with the file when given the address. With a map they hit missing pages less often.
- Most files on the web were made by plugins and platforms, not by owners.
- Google Search ignores the file.
## What is not proven
- An effect on rankings in any search engine.
- An effect on the number of mentions in AI answers.
- That any mass AI system looks for the file on other people's sites by itself. No bot owner has said so.
## What changed in the understanding
A year ago `llms.txt` was discussed as a promotion method. The data did not confirm it. Its real role became visible instead: a map for an agent in an environment where the agent already works.
It also became clear that the file is one layer of several. A machine has to get the page, learn where it may go, find the addresses, understand what is on the page and read the text. Structured data is responsible for understanding, and search engines say directly that they read it.
## What to do with a site
1. Make sure pages are reachable and serve text without scripts.
2. Sort out bots in `robots.txt`.
3. Add structured data: [what it is and why](https://llms-full-txt.ru/en/guides/json-ld/).
4. Make `llms.txt` if the site has something to show an agent: [getting started](https://llms-full-txt.ru/en/guides/getting-started/).
5. Check the result and watch the logs: [validation](https://llms-full-txt.ru/en/guides/validation/).
The whole path in order: [A site ready for AI](https://llms-full-txt.ru/en/guides/ai-ready-site/). The status of the proposal itself: [llms.txt status](https://llms-full-txt.ru/en/guides/standard-status/).
## Lighthouse 13.5: llms.txt becomes one of the discovery points
URL: https://llms-full-txt.ru/en/blog/lighthouse-13-5-agent-discovery/
Event date: 2026-09-18
Published: 2026-10-03
[Lighthouse 13.5.0](https://github.com/GoogleChrome/lighthouse/releases/tag/v13.5.0) came out on September 18, 2026. The `llms.txt` check neither disappeared nor grew stronger: it moved into a common group of agent discovery checks.
## What changed
The `llms.txt` line used to stand alone in the Agentic Browsing category. Now it sits in one group with other points through which an agent learns what a site can offer.
The meaning of the move: the file is treated not as a goal in itself, but as one of the ways of discovery.
## What stayed the same
- Only the response of `/llms.txt` is checked. The content is not judged.
- Status `404` gives "not applicable", not an error.
- The category is experimental and has no overall score.
- The performance, accessibility and SEO scores do not depend on the file.
## History of the check
| Date | Version | What happened |
|---|---|---|
| May 1, 2026 | 13.2.0 | the check was added to the code |
| May 7, 2026 | 13.3.0 | turned on by default |
| July 20, 2026 | 13.4.1 | available through the PageSpeed Insights API |
| September 18, 2026 | 13.5.0 | grouped with other discovery checks |
## How to read it
In four and a half months the check went from a novelty to a line in a group. The Chrome team neither removed it nor made it mandatory. The makeup of an experimental category changes from release to release, and plans should not be built on its current shape.
## What a site owner should do
If the file exists and is real, nothing needs to be done. If new lines appeared next to `llms.txt` in the report, read them as hints, not errors. What the check proves and what it does not: [Lighthouse and llms.txt](https://llms-full-txt.ru/en/guides/lighthouse/).
## Mintlify: agents request documentation twice as often as people
URL: https://llms-full-txt.ru/en/blog/mintlify-agents-outnumber-people/
Event date: 2026-09-16
Published: 2026-10-03
On September 16, 2026, [Mintlify](https://www.mintlify.com/blog/the-state-of-knowledge-2026-highlights) published a report on who reads documentation on its platform. The headline number: in August agents made almost twice as many requests as people.
The source is an interested party: Mintlify sells a documentation platform and creates files for agents itself. The numbers describe its customers, not the whole web.
## What is in the report
| Measure | Value |
|---|---|
| Agent requests, August 2026 | 257 million |
| Human page loads, August 2026 | 131 million |
| Share of agent requests through Markdown, `llms.txt` and skills | 83% |
The third row matters more than the first two. Most agent requests went not to regular HTML pages but along routes made for machines.
## How to read it
Developer documentation is the first environment where the machine reader became the main one. The reason is clear: a developer works in an editor with an assistant, and the agent reads the documentation for them.
Here a file for agents is part of the product, not an experiment.
## What the report does not show
- That shops, blogs and company sites look the same. In the logs of regular sites the picture is the opposite: according to [Ahrefs](https://ahrefs.com/blog/llmstxt-study), 97% of `llms.txt` files were requested by nobody.
- That agents come by themselves. In documentation, a person or a tool usually gives the agent the address.
- How requests are counted: one agent session can produce dozens of requests, while a person produces one page load.
## What a site owner should do
Decide which environment the site belongs to. If it is documentation or help, Markdown versions and `llms.txt` are worth doing now: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/). If it is a shop, start with page accessibility and product markup: [A site ready for AI](https://llms-full-txt.ru/en/guides/ai-ready-site/).
## IETF AIPREF: the AI preference vocabulary is still a draft
URL: https://llms-full-txt.ru/en/blog/ietf-aipref-drafts/
Event date: 2026-09-14
Published: 2026-10-03
In mid-September 2026 the IETF [AIPREF](https://datatracker.ietf.org/wg/aipref/about/) working group published another revision of its draft preference vocabulary. It is an attempt to do what `robots.txt` cannot: to say not "do not go here" but "do not use this in that way".
## Which problem it solves
`robots.txt` controls crawling. It does not tell why a bot came: for a page for a search index, for text to train a model, or for an answer to a user's question.
Today that difference is expressed by bot names: one bot of a company collects training data, another serves search. It works, but requires knowing every bot by name.
## What is in the drafts
The group is preparing two documents:
- **a vocabulary** - the names of kinds of use an owner can allow or forbid: model training, use in AI systems, search;
- **a way to attach it** - how to state these preferences: in the server response and next to the `robots.txt` rules.
## What state it is in
Both documents are working group drafts. There is no RFC. The milestone set for August 2026 passed without a published standard.
In parallel there is `Content-Signal`, a line that [Cloudflare documents](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) and adds to its managed `robots.txt`. It is one company's initiative, not a standard.
## What it does not mean
- That bots already respect these preferences. Bot owners have made no such commitment.
- That `robots.txt` is obsolete. It remains the only mechanism bot owners describe directly in their documentation.
## What a site owner should do
Nothing new for now. The working tool is the same: `robots.txt` rules for specific bots in three classes. The bot table checked against owner documentation: [AI crawlers and robots.txt](https://llms-full-txt.ru/en/guides/ai-crawlers/).
## Common Crawl analyzed 584 thousand llms.txt files
URL: https://llms-full-txt.ru/en/blog/common-crawl-584k-llms-txt/
Event date: 2026-08-31
Published: 2026-10-03
On August 31, 2026, [Common Crawl](https://commoncrawl.org/blog), the open web crawl archive, published an analysis of `llms.txt` files. Before it, people counted who has the file. Here, for the first time, someone looked inside.
## What was measured
The crawl found 584,107 files, and their content was analyzed.
| Measure | Value |
|---|---|
| Made by a generator, not by hand | 68% |
| Belong to Wix sites | 41% |
| Have the full structure | 50% |
| Contain no links at all | 23% |
| Successful responses that were an HTML page | 40% |
## Three findings
**People do not write the files.** Two thirds were made by a generator, most often a platform that switched the feature on for all its sites at once.
**Every fourth file is useless.** The job of the file is to show an agent where to go. A file without links does not do it.
**Status `200` proves nothing.** Many sites answer any unknown address with a success code. At `/llms.txt` they serve not a file but the HTML of the home page or an error page. A check that looks only at the status code will pass such a site.
## What the analysis does not show
- Who reads these files. It measures content, not requests.
- How useful the files with a correct structure are: formal completeness is not the same as a good choice of links.
## What a site owner should do
Check three things, a minute each:
1. `/llms.txt` serves text, not an HTML page.
2. The file has links, and they open.
3. If a plugin or a platform made the file, it has no drafts or service pages.
Commands for checking: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/). Typical errors and how to avoid them: [Best practices](https://llms-full-txt.ru/en/guides/best-practices/).
## The second edition of llms.txt is out
URL: https://llms-full-txt.ru/en/blog/llms-txt-v2/
Event date: 2026-08-10
Published: 2026-10-03
On August 10, 2026, Jeremy Howard published the second edition of the `llms.txt` proposal. The first one came out on September 3, 2024.
The main point: the file format barely changed. What changed is everything around it.
## What is new
**Discovery.** A page can now say where its Markdown version is and which `llms.txt` describes it:
```html
```
The same can be sent in the `Link` response header.
**Files in subpaths.** A file describes the addresses under its own path. If several files match, the agent takes the most specific one.
**Two forms of Markdown addresses.** Both `page.html.md` and `page.md` are allowed.
**How the file is used.** An agent scans the file or searches it, then follows the links.
## What was removed
The `llms_txt2ctx` tool and the `llms-ctx.txt` files. The special meaning of the `Optional` section went with them: it is now a convention for secondary links, not a command to a tool.
## What the edition does not contain
- `llms-full.txt` and `llms-small.txt`. They are an ecosystem convention; the proposal has no such files.
- An address in `/.well-known/`. It is rejected with an explanation: many authors control only a path on a shared host.
- The status of a standard. It is still a proposal by one author, not an IETF or W3C document.
## Does anything need to change
A file written by the first edition stays correct. Three things are worth doing:
1. Add a `rel="describedby"` link to the file on pages.
2. If Markdown versions exist, add `rel="alternate"`.
3. Do not count on the `Optional` section being handled in a special way.
The table of differences: [Changes in v2](https://llms-full-txt.ru/en/specification/changes/). The relations with server examples: [Discovery of the file and Markdown versions](https://llms-full-txt.ru/en/specification/discovery/).
## Mintlify test: with llms.txt an agent hits fewer missing pages
URL: https://llms-full-txt.ru/en/blog/mintlify-markdown-benchmark/
Event date: 2026-07-17
Published: 2026-10-03
On July 17, 2026, the documentation platform [Mintlify](https://www.mintlify.com/blog/llms-txt-agent-benchmark) published a comparison: how agents cope with questions about documentation when they get it in different formats.
The source is an interested party. Mintlify creates `llms.txt` and Markdown versions for its customers, and the test ran on sites of its own platform.
## What was compared
Four variants of the same documentation:
1. pages in HTML;
2. pages in Markdown;
3. Markdown with a link to `llms.txt`;
4. Markdown with the content of `llms.txt` inside the page.
Agents running OpenAI and Anthropic models were asked five questions on each of 20 sites, three times in each format. 2,400 runs in total.
## What came out
| Format | Requests to missing addresses per task |
|---|---|
| HTML | 2.23 |
| Markdown | 1.42 |
| Markdown and `llms.txt` | 0.11 |
Answer accuracy barely changed: in every format it stayed above 90%.
## How to read it
The file did not make answers more correct. It cut the wandering: with a map, an agent stops guessing addresses and follows links.
This matches how the second edition of the proposal describes the file: an agent scans `llms.txt` and follows the links it needs.
## What the test does not show
- That bots of AI search systems behave the same way. The test covered agents that were given a task and a site.
- That the result carries over to shops and blogs. All 20 sites are documentation on one platform.
- That a site will be mentioned in answers more often.
## What a site owner should do
If you run documentation, the benefit is measurable: fewer wasted agent requests. How to serve Markdown and connect it with the file: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/).
## Chrome describes how to prepare a site for agents
URL: https://llms-full-txt.ru/en/blog/chrome-agent-ready-toolkit/
Event date: 2026-06-22
Published: 2026-10-03
On June 22, 2026, the [Chrome developer blog](https://developer.chrome.com/blog/agent-ready-toolkit) published a post about a toolkit for sites that are ready to work with agents. It is the first coherent explanation from the browser team of what it means by such readiness.
## What is in the toolkit
- **The Agentic Browsing category in Lighthouse.** Checks of whether an agent can work with a page.
- **WebMCP.** A proposed standard by which a site describes the actions available to an agent.
- **Chrome DevTools for agents.** Debugging tools that an agent can use.
## What the checks look at
The post names three areas: accessibility, layout stability and the WebMCP connection.
The main idea is about accessibility. An agent in the browser reads a page through the accessibility tree, the same structure that screen readers use. A button without a name is invisible to both.
The category is called informational: it shows pass, fail and warnings, but gives no overall score.
## What the post does not contain
It does not mention `llms.txt`. The file check exists in Lighthouse, but when the Chrome team describes preparing a site, it talks about something else: the markup of a page and its behavior.
That is a useful correction to the talk about the file. For an agent that acts in a browser, a clear page matters more than a site map.
## What a site owner should do
Start with what also helps people: names on buttons and fields, headings in order, a layout that does not jump. The layers of preparing a site in order: [A site ready for AI](https://llms-full-txt.ru/en/guides/ai-ready-site/). What exactly Lighthouse checks about the file: [Lighthouse and llms.txt](https://llms-full-txt.ru/en/guides/lighthouse/).
## Ahrefs: 97% of llms.txt files were never requested
URL: https://llms-full-txt.ru/en/blog/ahrefs-llms-txt-study/
Event date: 2026-06-15
Published: 2026-10-03
On June 15, 2026, [Ahrefs](https://ahrefs.com/blog/llmstxt-study) published a study of requests to `llms.txt`. It is the largest measurement so far made from request logs rather than from the presence of the file.
## What was measured
The sample is 137,210 domains connected to Ahrefs web analytics. The period is May 2026. The authors looked at who has the file and who requested it.
| Measure | Value |
|---|---|
| Domains with the file | 28% |
| Files with no requests in a month | 97% |
| Requests made by bots | 96% |
| Share of SEO audit tools | 21.7% |
| Share of AI bots | 19.5% |
Among AI bots, agents made the most requests, and bots that collect answers to user questions made the fewest: about one percent.
## How to read it
**28% is an upper bound.** Ahrefs analytics is installed by people who work on promotion. A crawl of the whole web gives about two percent.
**"Requested" does not mean "used".** A log shows a request to an address. What the bot did with the content is not visible in the log.
**A notable share of requests comes from checking tools.** The file is requested by an audit more often than by an AI system.
## What the study does not show
- That the file is useless for agents in code editors: there the user gives the address, and that request is rare but purposeful.
- How the picture changes after the second edition of the proposal, which came out in August.
## What a site owner should do
Look in your own server logs to see who requests `/llms.txt`. It is the only measurement that concerns your site: [How to measure AI visibility](https://llms-full-txt.ru/en/guides/ai-visibility/). A summary of all studies: [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/).
## Google documents it: Search does not use llms.txt
URL: https://llms-full-txt.ru/en/blog/google-search-ignores-llms-txt/
Event date: 2026-06-15
Published: 2026-10-03
On June 15, 2026, [Google extended its guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) for site owners on AI features in Search. It now has a direct statement about machine-readable files such as `llms.txt`.
## What it says
The meaning of two sentences in the guide:
- there is no need to create new machine-readable files for AI features in Search, because Search does not use them;
- such files will neither help nor harm, because Search ignores them.
Before that, Google's position was known from remarks by its staff on social networks and at conferences. Now it is written in the official documentation.
## What it covers
Google Search only: regular results, AI Overviews and AI Mode. For them, according to the same guide, regular SEO works: accessible pages, useful content, structured data.
## What it does not cover
- Agents in code editors and developer tools. They read `llms.txt` when they are given the address.
- Other search and AI systems. Google does not speak for them.
- Chrome. Lighthouse has been checking for the file since May 2026, and there is no contradiction: Lighthouse judges whether a site is convenient for an agent, not how results are built.
## What follows
The file is not worth making for Google rankings. It is worth making if agents read the site: in documentation, help, a catalog with a clear structure.
For visibility in AI features of Search, what Google reads and writes about directly matters more: [structured data](https://llms-full-txt.ru/en/guides/json-ld/). All studies and statements in one place: [Does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/).
## Lighthouse starts checking llms.txt
URL: https://llms-full-txt.ru/en/blog/lighthouse-checks-llms-txt/
Event date: 2026-05-07
Published: 2026-10-03
[Lighthouse 13.3.0](https://github.com/GoogleChrome/lighthouse/releases/tag/v13.3.0) came out on May 7, 2026. In it the `llms.txt` check is on by default in the experimental Agentic Browsing category. A week earlier, in version 13.2.0, the check appeared in the code.
Lighthouse is the Chrome team's tool for checking page speed, accessibility and SEO. Its report now has a line about a file for agents.
## What is checked
One fact: whether `/llms.txt` answers.
| Site response | What the report shows |
|---|---|
| The file is served | the check passes |
| Status `404` | "not applicable" |
| Server error | a warning |
Lighthouse does not read the content of the file. An empty file and a thought-out map are equal for the check.
## Why it is a notable event
Before May 2026 no mainstream tool for site owners mentioned `llms.txt`. Now everyone who opens a Lighthouse report learns about the file.
## What the check does not mean
- It is not a signal for Google Search. A month later [Google wrote](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) in its documentation that Search does not use such files.
- A missing file does not lower the performance, accessibility or SEO scores.
- The category is experimental and has no overall score.
## What a site owner should do
Creating an empty file for the sake of a line in a report is not worth it: that is how files without a single link appear. If the site has documentation, a catalog or a knowledge base, make a real file and the check will pass by itself. The detailed breakdown: [Lighthouse and llms.txt](https://llms-full-txt.ru/en/guides/lighthouse/).
## llms.txt in 2026: From Experiment to Standard
URL: https://llms-full-txt.ru/en/blog/llms-txt-2026/
Published: 2026-03-01
> **Update, October 3, 2026.** This post is kept as published on March 1. The picture has since become clearer: llms.txt remains a proposal by a single author, not a standard; v2 was published on August 10, 2026; there is no data showing the file affects search or citations, and the 10% share comes from one study that other measurements do not confirm. Current facts: [Status of llms.txt](https://llms-full-txt.ru/en/guides/standard-status/) and [Changes in v2](https://llms-full-txt.ru/en/specification/changes/).
## From Idea to Adoption
In September 2024, Jeremy Howard (Answer.AI / fast.ai) proposed a simple idea: a markdown file at the site root describing content structure for language models. The idea resonated.
By March 2026, llms.txt is used by Anthropic, Cloudflare, Stripe, Vercel, and thousands of other sites. Adoption exceeded 10% among 300,000 analyzed domains.
## Key Milestones
- **September 2024** — first specification published
- **December 2024** — 500+ sites with llms.txt
- **March 2025** — generators available for all popular CMS
- **2026** — 10%+ adoption, integration with IDEs and AI assistants
## What's Next?
Three development directions: formal standardization, MCP integration, and specialized extensions for different content types.
## AI crawlers guide: who to allow, who to block
URL: https://llms-full-txt.ru/en/blog/ai-crawlers-guide/
Published: 2026-02-15
> **Update, October 3, 2026.** This post is kept as published on February 15, and its table is inaccurate. `GPTBot` and `ClaudeBot` cover training only; search and user requests are served by other bots. No bot owner states that it reads other sites' llms.txt, so the file should not be expected to raise visibility in AI answers. The table checked against the owners' documentation: [AI crawlers and robots.txt](https://llms-full-txt.ru/en/guides/ai-crawlers/).
## AI Crawlers in 2026
The AI crawler landscape has changed significantly. Today, more than 15 different bots regularly scan websites for model training and inference.
## Main AI Crawlers
| Bot | Company | Purpose |
|-----|---------|---------|
| GPTBot | OpenAI | Training + inference |
| ClaudeBot | Anthropic | Training + inference |
| Google-Extended | Google | Training Gemini |
| PerplexityBot | Perplexity | Search index |
## Recommendations
For documentation sites, allowing all AI crawlers is recommended — it increases your content visibility in AI responses. The llms.txt file additionally helps bots find structured information.
## Cloudflare serves Markdown to agents instead of HTML
URL: https://llms-full-txt.ru/en/blog/cloudflare-markdown-for-agents/
Event date: 2026-02-12
Published: 2026-10-03
On February 12, 2026, Cloudflare announced [Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/). A site behind Cloudflare can serve an agent a Markdown version of a page without changing a line of its own code.
## How it works
The agent adds the `Accept: text/markdown` header to its request. Cloudflare's network fetches the regular HTML page from the site, converts it to Markdown and returns the result.
```http
GET /docs/install/ HTTP/1.1
Host: example.com
Accept: text/markdown
```
The response carries the `text/markdown` type and an `x-markdown-tokens` header with an estimate of the text size in tokens. A browser does not send that header and gets HTML as before.
At the time of the announcement the feature was in beta and was enabled by the site owner in the dashboard on paid plans.
## How much it saves
The announcement gives Cloudflare's own example: the text of the post takes 16,180 tokens in HTML and 3,150 in Markdown, which is 80% less. This is a measurement of one page made by the author of the feature, not a study.
## What it does not change
- The conversion is automatic. If the HTML has no text because scripts draw it later, the Markdown will not have it either.
- The agent has to ask for Markdown. Those that do not send the header get HTML.
- There is no data showing that the feature increases mentions of a site in AI answers.
## What a site owner should do
If the site is already behind Cloudflare, the feature can be enabled and checked with one request. If not, a separate `.md` address or a `rel="alternate"` link gives the same result. Three ways with examples and checks: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/).
## Web Almanac: two sites in a hundred have llms.txt
URL: https://llms-full-txt.ru/en/blog/web-almanac-2025-llms-txt/
Event date: 2026-01-15
Published: 2026-10-03
[Web Almanac](https://almanac.httparchive.org/en/2025/seo) is a yearly report on the state of the web built on the HTTP Archive crawl. Its SEO chapter for 2025, published on January 15, 2026, counted `llms.txt` files for the first time.
## What was measured
The HTTP Archive crawl requests the home pages of millions of sites. For each site the authors checked whether `/llms.txt` answers.
| Measure | Value |
|---|---|
| Sites with the file, desktop sample | 2.13% |
| Sites with the file, mobile sample | 2.10% |
| Files found | 324,184 |
| Created by the All in One SEO plugin | 39.6% |
| Created by the Yoast SEO plugin | 3.6% |
## What follows
Two percent is the most sober estimate available. It comes from a broad crawl, not from a sample of sites whose owners work on promotion. In such samples the share is several times higher: [Ahrefs](https://ahrefs.com/blog/llmstxt-study) got 28% and called it an upper bound.
The second finding matters more than the first. Almost half of the files were created by two WordPress plugins. A file appears not because the owner decided to write it, but because the feature is switched on in a plugin.
## What the report does not show
- Who reads these files. It counts presence, not requests.
- How good the files are. The content was not analyzed; [Common Crawl](https://commoncrawl.org/blog) did that later.
- What happens below the home page: files in subpaths were not counted.
## What a site owner should do
If the site runs on WordPress, check whether a plugin has published a file without your knowledge and what it says: [llms.txt in WordPress](https://llms-full-txt.ru/en/guides/wordpress/). All adoption estimates, with an explanation of why they differ: [Who publishes llms.txt](https://llms-full-txt.ru/en/reference/adoption/).