Common Crawl analyzed 584 thousand llms.txt files
The first analysis of llms.txt content from an open web crawl - 68% of files made by a generator, 23% without a single link, and 40% of successful responses turned out to be an HTML page
On August 31, 2026, Common Crawl, the open web crawl archive, published an analysis of llms.txt files. Before it, people counted who has the file. Here, for the first time, someone looked inside.
What was measured
The crawl found 584,107 files, and their content was analyzed.
| Measure | Value |
|---|---|
| Made by a generator, not by hand | 68% |
| Belong to Wix sites | 41% |
| Have the full structure | 50% |
| Contain no links at all | 23% |
| Successful responses that were an HTML page | 40% |
Three findings
People do not write the files. Two thirds were made by a generator, most often a platform that switched the feature on for all its sites at once.
Every fourth file is useless. The job of the file is to show an agent where to go. A file without links does not do it.
Status 200 proves nothing. Many sites answer any unknown address with a success code. At /llms.txt they serve not a file but the HTML of the home page or an error page. A check that looks only at the status code will pass such a site.
What the analysis does not show
- Who reads these files. It measures content, not requests.
- How useful the files with a correct structure are: formal completeness is not the same as a good choice of links.
What a site owner should do
Check three things, a minute each:
/llms.txtserves text, not an HTML page.- The file has links, and they open.
- If a plugin or a platform made the file, it has no drafts or service pages.
Commands for checking: Validating files and markup. Typical errors and how to avoid them: Best practices.