Does llms.txt work
What the Ahrefs, SE Ranking and Common Crawl studies and Google's documentation show - what is confirmed about llms.txt, what is not proven, and what does not follow
The key numbers
Section titled “The key numbers”What the studies say
Section titled “What the studies say”| Who and when | What was measured | Result | What it does not show |
|---|---|---|---|
| SE Ranking, November 7, 2025 | about 300 thousand domains: presence of the file and how often AI systems cite the domain | 10% of domains have the file; in a citation prediction model the “has llms.txt” feature turned out to be noise | causation; the authors call the findings dataset-dependent |
| Ahrefs, June 15, 2026 | 137,210 domains, request logs for May 2026 | 97% of files got zero requests; 96% of requests came from bots, AI bots made 19.5% | “requested” is not “used”; the sample leans toward an SEO audience |
| Longato, June 1, 2026 | sites on one platform, 30 days of logs | of 22,494 requests for the file, 258 came from verified AI systems, that is 1.1% | one platform |
| OtterlyAI, February 2026 | one site, 90 days, AI agents only | llms.txt received about 0.1% of agent visits | one site |
| Common Crawl, August 31, 2026 | 584,107 files from an open crawl | 68% made by a generator, 23% without links, 40% of successful responses were an HTML page | who reads these files |
| Mintlify, September 16, 2026 | documentation sites on its own platform | agents request documentation more often than people | an interested source: the platform generates such files itself |
What Google says
Section titled “What Google says”Google’s documentation for site owners is direct: there is no need to create new machine-readable files for the AI features of Search, because Search does not use them; they will neither harm nor help.
The statement covers Google Search only. Meanwhile Lighthouse, a tool from the same company, checks for llms.txt in the experimental Agentic Browsing category. There is no contradiction: Search ranks pages, Lighthouse judges how well a site works with an agent. More: the Lighthouse audit.
What AI bot owners say
Section titled “What AI bot owners say”OpenAI, Anthropic and Perplexity publish llms.txt in their own documentation. None of them claims that its bot reads such files on other sites. Their bot descriptions do not mention the file.
Publishing your own file and reading other people’s are different things.
The bottom line
Section titled “The bottom line”What is known and what is not
Confirmed
- Large documentation sites publish the file, including the makers of AI systems.
- Agents and assistants in code editors work with the file when given its address.
- Most published files are generated automatically and hardly anyone requests them.
- Google Search ignores the file.
Not proven
- An effect on rankings in any search engine.
- An effect on mentions and citations in AI answers.
- That any mass-market AI system looks for
llms.txton sites by itself.
What it does not mean
- Low traffic to the file does not make it harmful. It is simply rarely needed.
- No link to citations today does not mean never: agent behavior keeps changing.
- Results for documentation sites do not transfer to stores, and the other way round.
llms.txt is a map for an agent, not a promotion lever.
Dated notes in the blog: the Ahrefs study, Google’s position, the Common Crawl analysis.
What to do with this
Section titled “What to do with this”- A documentation site. Make the file: it is cheap, and agents in this environment use it. Getting started.
- A store or a company site. You may make the file, but do not start with it. Page delivery, product markup and clear text come first: An AI-ready site.
- You want to measure it yourself. Look in the server logs for who requests
/llms.txt: How to measure AI visibility.
Who publishes the file and what exactly: the adoption registry. The status of the proposal: llms.txt status.