# Common Crawl analyzed 584 thousand llms.txt files

> The first analysis of llms.txt content from an open web crawl - 68% of files made by a generator, 23% without a single link, and 40% of successful responses turned out to be an HTML page

Page: https://llms-full-txt.ru/en/blog/common-crawl-584k-llms-txt/
Site index: https://llms-full-txt.ru/en/llms.txt

Event date: 2026-08-31

Published: 2026-10-03

On August 31, 2026, [Common Crawl](https://commoncrawl.org/blog), the open web crawl archive, published an analysis of `llms.txt` files. Before it, people counted who has the file. Here, for the first time, someone looked inside.

## What was measured

The crawl found 584,107 files, and their content was analyzed.

| Measure | Value |
|---|---|
| Made by a generator, not by hand | 68% |
| Belong to Wix sites | 41% |
| Have the full structure | 50% |
| Contain no links at all | 23% |
| Successful responses that were an HTML page | 40% |

## Three findings

**People do not write the files.** Two thirds were made by a generator, most often a platform that switched the feature on for all its sites at once.

**Every fourth file is useless.** The job of the file is to show an agent where to go. A file without links does not do it.

**Status `200` proves nothing.** Many sites answer any unknown address with a success code. At `/llms.txt` they serve not a file but the HTML of the home page or an error page. A check that looks only at the status code will pass such a site.

## What the analysis does not show

- Who reads these files. It measures content, not requests.
- How useful the files with a correct structure are: formal completeness is not the same as a good choice of links.

## What a site owner should do

Check three things, a minute each:

1. `/llms.txt` serves text, not an HTML page.
2. The file has links, and they open.
3. If a plugin or a platform made the file, it has no drafts or service pages.

Commands for checking: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/). Typical errors and how to avoid them: [Best practices](https://llms-full-txt.ru/en/guides/best-practices/).