Skip to content
LLMs Full Text
Select theme

Markdown versions of pages

Why a site may serve Markdown versions of its pages to AI agents, three ways to do it - a separate .md address, the Accept header and a rel=alternate link - and what is really known about the benefit

HTML and Markdown of one page

Section titled “HTML and Markdown of one page”
What a client receives

HTML

for browsers and search

  • Menus, header, footer, scripts, styles
  • Structured data in the page code
  • A lot of service markup around the text

Markdown

for an agent

  • Content only
  • Headings and lists keep the structure
  • No JSON-LD and no interactive parts

Three ways to serve Markdown

Section titled “Three ways to serve Markdown”

.md is appended to the page address. The second edition of the llms.txt proposal describes this.

https://example.com/docs/install.html      the page
https://example.com/docs/install.html.md   its Markdown version


https://example.com/docs/                  a page without a file name
https://example.com/docs/index.html.md     its Markdown version

Both forms are allowed: page.html.md and page.md. The advantage: the address opens in a browser and can be listed in llms.txt.

The ways do not exclude one another. More on discovery relations: Discovery of the file and Markdown versions.

What a Markdown version should contain

Section titled “What a Markdown version should contain”
  • The same content as the HTML. Different text for machines is cloaking, and search engines treat it badly.
  • One first-level heading and meaningful section headings.
  • Absolute links. An agent reads the file outside the site and has nothing to resolve a relative path against.
  • Tables as tables, code in code blocks. A machine cannot read a screenshot of a table.
  • No navigation, footer or forms.
  • Content type text/markdown or text/plain, UTF-8.

How it relates to llms.txt

Section titled “How it relates to llms.txt”

llms.txt should link to Markdown versions when they exist. Then an agent walks the whole path: the map, then clean text.

Without Markdown versions, a file linking to ordinary pages is still valid. The proposal recommends Markdown and does not require it.

What is known about the benefit

Section titled “What is known about the benefit”

What is known and what is not

Confirmed

  • The documentation sites of OpenAI, Anthropic, Stripe, Cloudflare and GitHub serve Markdown versions of pages.
  • Documentation platforms generate them automatically for every hosted site.
  • Mintlify reports that on its sites agents request documentation more often than people. The source has an interest: it is its product.

Not proven

  • That a Markdown version increases citations or referrals.
  • That the bots of AI search systems request .md by themselves. In an OtterlyAI experiment there was not a single such request in 14 days.

What it does not mean

  • It is not a way into a search index: search reads HTML.
  • It does not replace structured data: Markdown carries no entity markup.

Large documentation sites publish Markdown versions. Independent measurements of the benefit are still scarce.

The sober conclusion: Markdown versions help where agents read a site while working, above all in documentation. For a store or a blog it is groundwork, not an urgent task.

More in the blog: Cloudflare serves Markdown to agents, the Mintlify test of four formats.

How to check

Section titled “How to check”
  1. Request the page with Accept: text/markdown, or open the address ending in .md.
  2. The response is 200, typed text/markdown or text/plain.
  3. The body is text, not an HTML page.
  4. The content matches the ordinary page.
  5. Links are absolute and working.
  6. When rel="alternate" is present, its address answers 200.
Окно терминала
curl -sI -H 'Accept: text/markdown' https://example.com/docs/install/
curl -sI https://example.com/docs/install.html.md

Who does this

Section titled “Who does this”

  • OpenAI

    developers.openai.com/llms.txt
    llms-full.txt
    yes
    Markdown pages
    via the Accept header

    What it shows. A root index plus nested files for /api, /codex, /commerce

    What it does not prove. That ChatGPT reads other sites’ llms.txt

  • Anthropic

    platform.claude.com/llms.txt
    llms-full.txt
    yes
    Markdown pages
    yes, with rel=alternate

    What it shows. A docs index and the full text in one file

    What it does not prove. That Claude reads other sites’ llms.txt

  • Cloudflare

    developers.cloudflare.com/llms.txt
    llms-full.txt
    yes
    Markdown pages
    yes

    What it shows. A very large index: the full text takes 67 MB

    What it does not prove. That a file this big is usable by an agent as a whole

  • Stripe

    docs.stripe.com/llms.txt
    llms-full.txt
    no
    Markdown pages
    via the Accept header

    What it shows. llms-full.txt is not required: an index and Markdown pages are enough

    What it does not prove. That skipping the full file is a general rule

  • Cursor

    cursor.com/llms.txt
    llms-full.txt
    not checked
    Markdown pages
    via the Accept header

    What it shows. The only measured site with rel=describedby from the second edition

    What it does not prove. That the second edition is widely adopted

  • GitHub Docs

    docs.github.com/llms.txt
    llms-full.txt
    no
    Markdown pages
    yes

    What it shows. An index without a full file on large docs

    What it does not prove. That Copilot relies on this file