Make your site clear to neural networks and AI agents
Search is changing: more and more people get the answer right inside ChatGPT, Gemini and other AI systems. This reference explains how to prepare a site for that world - from content structure and Markdown to llms.txt, crawlers and GEO.
- A personasks a question
- AI search or an agentlooks for a source
- Your siteserves content in a clear formHTMLMarkdownllms.txtstructured data
- The answerwith a link to the source
Your site is now read not only by people
Before
- A persontypes a query
- Searchshows a list of links
- A sitethe person reads the page
Now
- A personasks a question
- AI search or an agentfinds and reads pages by itself
- Sourcesseveral sites at once, yours among them
- A ready answerthe person may not open a single page
A site used to be prepared for a browser and a search robot first. Now there is one more consumer: the AI agent. It also has to find the right page, understand its structure and get the content without noise.
Where is llms.txt here?
A map of material for an AI agent
llms.txt is a file in the site root: the name, a short description and lists of links to what matters.
File formatThree files, three different questions
robots.txtwhere a robot may gositemap.xmlwhich pages existllms.txtwhich material may be useful to an agent
This is a simplified model, not three interchangeable standards.
One file is not enough to make a site ready for AI
A site ready for AI
Five parts of one task. llms.txt covers one of them.
- FindCrawlers, robots.txt, links, indexability.
- UnderstandClear page structure, headings, Markdown.
- Choosellms.txt, docs navigation, files in subpaths.
- TrustSources, authorship, freshness, evidence.
- MeasureBot logs, AI referrals, mentions and citations.
Does it help promotion at all?
What is known and what is not
Confirmed
- Large documentation sites publish llms.txt themselves.
- Agents do work with Markdown versions of pages.
- Lighthouse checks the file in the experimental Agentic Browsing category.
- AI bots visit sites: their owners describe it in their own documentation.
Not proven
- That having llms.txt improves search rankings.
- That the file guarantees a place in a ChatGPT answer.
- That it increases citations by itself.
llms.txt is one part of preparing a site for AI, not a magic SEO button.
What “promotion in neural networks” means in practice
- GEOGenerative Engine Optimization
- AEOAnswer Engine Optimization
- AI SEOSEO for AI search
- AI visibilitybeing visible in AI
One task
To be found, understood and used by an AI system
- Accessibilitythe page can be fetched
- Understandingthe structure can be parsed
- Citable contentthere is something to quote
- Measurementyou can see what happens
Many terms, one meaning. Nobody can promise a place in an AI answer.
A site can be prepared step by step
Check access
AI bots can fetch the pages of the site.
AI crawlersTidy the content
Clear headings and page structure.
Best practicesAdd Markdown
Where it is justified: documentation, help.
Markdown versions of pagesCreate llms.txt
A correct file by the second edition.
Getting startedCheck the file and links
Status code, structure, live addresses.
ChecklistMeasure the result
Bot logs, AI referrals, mentions and citations.
Measurement
You do not have to assemble everything by hand
For some CMS there are ready-made extensions: they collect the site structure, build llms.txt and keep it up to date.
- CMSpages, products, posts
- Extensionreads what is published
- Site structuresections and links
- llms.txtbuilt and kept fresh
- Checkthe same as for a hand-written file
Webasyst
Site
A map of site pages
- Data source
- Published site pages, the site name and contacts
- Creates
llms.txtllms-full.txt
Webasyst
Shop-Script
A map of an online store
- Data source
- Categories, products, brands, information pages and contacts
- Creates
llms.txtllms-full.txt
Webasyst
Blog
A map of blog posts
- Data source
- Published posts, categories and tags
- Creates
llms.txtllms-full.txt
Webasyst
Hub
A map of community discussions
- Data source
- Public sections and published discussions
- Creates
llms.txtllms-full.txt
Webasyst
Helpdesk
A map of a knowledge base
- Data source
- Public FAQ and pages; tickets and correspondence are left out
- Creates
llms.txtllms-full.txt
Webasyst
Photos
A map of photo albums
- Data source
- Public albums and pages
- Creates
llms.txtllms-full.txt
Promo code for a discount on your first Webasyst order:BE462B3B6D
Commercial extensions from the Webasyst Store. Each one builds llms.txt and llms-full.txt from the data of its app and rebuilds them on a schedule or on demand. A file made by an extension is checked the same way as one written by hand.
Where to start for you
Site owner
«I need to understand whether I should do anything at all»
SEO specialist and marketer
«I want to understand GEO, AI crawlers and measurement»
Developer
«I want to implement it correctly»
It is already used in real projects
These companies publish llms.txt or Markdown representations in their own documentation. It does not mean their agents read your file.
- llms.txt files found in an open web archive
- 584K
- Common Crawl, August 2026
- of sampled sites publish the file
- 2.1%
- Web Almanac 2025
- the second edition of the proposal
- v2
- llmstxt.org, August 10, 2026
See how it works
- llms.txt in five minutes
A minimal file, where to put it and how to check it.
- Write the file
- Put it in the site root
- Check the server response
- llms.txt and llms-full.txt
An index and the full text: the difference.
llms.txtan index of linksllms-full.txtall text in one file - AI crawlers
Three kinds of bots, three different robots.txt decisions.
trainingsearchuser request
- llms.txt and MCP
Two independent systems and a bridge.
a filea protocol
- Real files
How OpenAI, Stripe and others build theirs.
- Markdown for agents
A clean version of a page at the same address.
- Discovery in v2
The alternate and describedby relations.
- Does llms.txt work
Studies and their limits.
What is changing right now
Lighthouse 13.5.0 grouped the llms.txt check with other agent discovery points
For a site owner. The check stays in the report, now next to other ways an agent discovers a site.
Common Crawl analyzed 584 thousand files: 68% made by a generator, 23% without a single link
For a site owner. A file without links is useless: check the content, not just the presence.
The second edition of the proposal: discovery relations and files in subpaths
For a site owner. The file can be declared in the page code, and sections can have files of their own.
Ahrefs: 97% of the files were never requested
For a site owner. The file does not bring bots by itself: read the server logs.
Google documented it: Search does not use such files
For a site owner. Do not expect an effect on Google rankings.
Lighthouse 13.3.0 started checking llms.txt by default
For a site owner. A missing file became visible in a regular page report.
Need to prepare an existing site?
Audit and implementation are done by the kaktak.net team.