Validating llms.txt and markup

How to check llms.txt, llms-full.txt, Markdown versions of pages and structured data - status code, content type, structure, links, the Google and Schema.org validators and typical mistakes

Why this is not a formality

Common Crawl analyzed 584 thousand llms.txt files found on the web. A large share of them does not work as intended.

of successful responses at /llms.txt were an HTML page: 40%; Common Crawl, Aug 31, 2026
of files contain no links at all: 23%; Common Crawl, Aug 31, 2026
of files have the full structure: title, summary, sections: 50%; Common Crawl, Aug 31, 2026

The most common failure is invisible to the owner: the site answers /llms.txt with 200 but serves the home page or a “not found” page.

Checking llms.txt

https://site/llms.txt answers 200 with no redirect chain.
The content type is text/plain or text/markdown, UTF-8.
The body is text, not HTML. The first line starts with # .
There is exactly one first-level heading.
A short summary follows the heading; the line starts with > .
Links are written as - [Title](address): note.
Link addresses are absolute and answer 200.
The file lists no private pages, drafts or addresses with parameters.
The file stays a reasonable size: it is a map, not the whole site.
robots.txt does not block the file from the robots it is meant for.

curl -sI https://example.com/llms.txt
curl -s https://example.com/llms.txt | head -n 5

In the first command look at the HTTP and content-type lines. In the second, the first line has to start with a hash sign.

Check every link in the file at once:

curl -s https://example.com/llms.txt \
  | grep -oE '\]\(https?://[^)]+\)' | tr -d '()]' \
  | while read -r url; do
      printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "$url")" "$url"
    done

Format rules: the file format, sections.

Checking llms-full.txt

It answers 200 with a text content type.
It holds the text of pages, not a list of links.
Its size is known. An agent will not read a file of tens of megabytes in one go.
It is built from the same content as the site and refreshed together with it.

curl -sI https://example.com/llms-full.txt | grep -i content-length

When this file is needed and when it is not: What is llms-full.txt.

Checking Markdown versions and relations

curl -sI -H 'Accept: text/markdown' https://example.com/docs/install/
curl -sI https://example.com/docs/install/ | grep -i '^link'
curl -s https://example.com/docs/install/ | grep -iE 'rel="(alternate|describedby)"'

A request with Accept: text/markdown, or the address ending in .md, returns text, not HTML.
The address in rel="alternate" answers 200.
The address in rel="describedby" points to an existing llms.txt.

More: Markdown versions of pages, discovery.

Checking structured data

Tool	What it checks	Address
Rich Results Test	which rich results Google can show, errors and warnings	`search.google.com/test/rich-results`
Schema Markup Validator	conformance to the Schema.org vocabulary, independent of Google	`validator.schema.org`
Yandex structured data validator	how Yandex sees the markup	`webmaster.yandex.ru/site/tools/microtest/`
Search Console	markup errors across the whole site over time	enhancement reports

The application/ld+json block is in the server response, not added later by scripts.
The JSON is valid: the tool opens it without a parse error.
The type matches the page: a product is Product, an article is BlogPosting or Article.
Required properties are filled. For a product: name, offer, price, currency, availability.
Values match the visible page: price, availability, author, date.
Each entity is described once, with no duplicate from the theme.
Breadcrumbs in the markup match the breadcrumbs on the page.
Pages of different kinds were checked: a product with variants, an out-of-stock product, an article, the home page.

curl -s https://example.com/product/kettle/ \
  | grep -c 'application/ld+json'

Zero means the server response has no markup. More than one is a reason to look for duplicated entities.

Which types are needed and what goes in them: Structured data and JSON-LD.

Typical mistakes

When to use: /llms.txt answers 200, but it is HTML
Trap: The site serves the home page or an error page for any unknown address. Lighthouse and an agent see a file that does not exist.
Seen in the wild: The content type is text/html and the first line starts with a tag.

When to use: Links in the file are relative or point to old addresses
Trap: An agent reads the file outside the site and cannot build the address. Some links answer 404.
Seen in the wild: The list has paths without a domain, or addresses that redirect.

When to use: The price in the markup differs from the price on the page
Trap: The theme prints markup from one field while a discount module shows the on-screen price from another.
Seen in the wild: The validator shows no errors: it checks form, not truth.

When to use: The file was created once and never refreshed
Trap: The generator was run by hand. The site changed, the map did not.
Seen in the wild: The file lists pages that are gone and misses new sections.

What a check does not show

A validator confirms form. It does not know whether the price is right or whether anyone reads the file.

Whether the file is actually read shows only in server logs: How to measure AI visibility.

Lighthouse checks llms.txt too, but only the fact of a response: Lighthouse and llms.txt.