llms.txt and AI search blog
The chronicle of the ecosystem: what happened, what was measured and what it means for a site owner. Notes are ordered by the date of the event. Evergreen material lives in the guides, and every note links to the page that is kept up to date.
Chronicle
llms.txt in autumn 2026: what is known
A summary of the year - the second edition of the proposal, the Lighthouse check, Google’s position, studies of requests and file quality. What is confirmed, what is not proven and what to do with a site
Lighthouse 13.5: llms.txt becomes one of the discovery points
In Lighthouse 13.5.0 the llms.txt check is grouped with other ways an agent discovers a site - what changed in the report and what stayed the same
Mintlify: agents request documentation twice as often as people
A documentation platform’s report for August 2026 - 257 million agent requests against 131 million human page loads. What it says about documentation and what it does not say about other sites
IETF AIPREF: the AI preference vocabulary is still a draft
The IETF AIPREF working group updated the draft vocabulary a site can use to say whether its content may be used for training and answers - what is in it and why it is too early to rely on it
Common Crawl analyzed 584 thousand llms.txt files
The first analysis of llms.txt content from an open web crawl - 68% of files made by a generator, 23% without a single link, and 40% of successful responses turned out to be an HTML page
The second edition of llms.txt is out
On August 10, 2026, the second edition of the llms.txt proposal was published - discovery relations, files in subpaths, two forms of Markdown addresses and a new role for the Optional section
Mintlify test: with llms.txt an agent hits fewer missing pages
Mintlify compared four documentation formats on 20 sites and 2,400 runs - what changed, what stayed the same and why the source has to be treated as an interested party
Chrome describes how to prepare a site for agents
The Chrome blog published an agent-ready toolkit - the Agentic Browsing category in Lighthouse, WebMCP and DevTools for agents. The post does not mention llms.txt
Ahrefs: 97% of llms.txt files were never requested
The Ahrefs study of 137 thousand domains - how many sites publish llms.txt, who requests it and why the numbers need to be read with caveats
Google documents it: Search does not use llms.txt
What exactly the Google guide on AI features in Search says, what it covers and what it does not, and how it fits with the Lighthouse check
Lighthouse starts checking llms.txt
In Lighthouse 13.3.0 the llms.txt check is on by default - what exactly it looks at, why an empty file passes and what it does not prove
llms.txt in 2026: From Experiment to Standard
How llms.txt evolved from Jeremy Howard's idea to 10% adoption among 300K domains. What changed and what's next.
AI crawlers guide: who to allow, who to block
Complete guide to AI crawlers in 2026 — GPTBot, ClaudeBot, Google-Extended. How to configure robots.txt for AI bots.
Cloudflare serves Markdown to agents instead of HTML
Cloudflare turned on page-to-Markdown conversion by the Accept header - how it works, how much it saves and what stays on the site owner’s side
Web Almanac: two sites in a hundred have llms.txt
The SEO chapter of Web Almanac 2025 counted llms.txt across the HTTP Archive crawl for the first time - 2.1% of sites, and most files come from WordPress plugins