An AI-ready site

1. 1robots.txtMay I go hereaccess rules for robots
2. 2sitemap.xmlWhich addresses existthe list of pages
3. 3llms.txtWhich material is most useful, and where it isa map for an AI agent
4. 4JSON-LDWhat is on this pagethe entity, its properties and relations
5. 5HTML / MarkdownWhat does it saythe content itself

This is a teaching model of roles. It does not claim that every system uses every layer the same way.

The layers do not replace one another. robots.txt says nothing about what is useful. llms.txt forbids nothing. Structured data does not replace the text of a page. And the most detailed llms.txt will not help a site that answers with an error.

Before any file comes a plain requirement: the server has to answer.

• The page returns 200, not a redirect chain and not an error.
• The main text is in the HTML sent by the server. If content appears only after scripts run in a browser, some automated clients will never see it.
• The response is fast and stable. A fast, reliable response lowers friction for every client: a person, a search robot, an agent.
• Reading the text requires no clicks, no dismissed dialogs and no scrolling.
• A page has one permanent address, and links are ordinary links that can be followed.

Structured data and llms.txt do not fix a slow or unreachable site. They describe what can already be fetched.

robots.txt holds access rules. The site owner tells robots where they may go.

AI companies run robots of three different classes: some collect training data, some build a search index, some come on behalf of a specific user. Blocking training while staying in AI search takes different lines in the file.

What to check: the file returns 200, the robots you need are not blocked by accident, and the decision for each class is deliberate.

More: AI crawlers and robots.txt.

sitemap.xml lists the pages of a site and helps discover them. It is a complete list without judgment: the sitemap does not say what matters.

sitemap.xml and llms.txt do not compete. The first answers “what exists”, the second “where to start”.

What to check: the sitemap exists, is referenced from robots.txt, contains only working addresses and no pages closed to indexing.

llms.txt is a map of useful material for an AI agent: the site name, a short description and lists of links with notes.

The caveats matter:

• it is a proposal, not an adopted standard;
• search engines do not use it for ranking;
• the second edition is current: it added discovery relations and files for separate sections of a site;
• llms-full.txt is not part of the proposal; it is an ecosystem convention.

What to check: the file sits in the site root, is served as text with 200, and its links work.

More: What is llms.txt, the file format, what is llms-full.txt.

llms.txt helps a machine decide where to go. Structured data states explicitly where it has arrived: this is a product, an article or a discussion, what it is called, who wrote it, what it costs, whether it is in stock.

The vocabulary for such a description is Schema.org. The most convenient way to write it is JSON-LD: a block of data in the page code, separate from the visible text.

1. llms.txt/products/widget/there is a useful page about a product here
2. JSON-LDthis is a Product: name, SKU, price, availability
3. HTML / Markdowndescription, specifications, instructions

For search engines this is a working layer, not decoration. Google states that it uses structured data to understand the content of a page. Yandex reads product markup and, for product pages marked up as Product with an Offer, gives the data on the page the highest priority over other sources.

If Schema.org can describe the kind of page and the markup is missing, the machine-readable form of the page is unfinished.

More: Structured data and JSON-LD.

Everything else leads here. A machine needs text: clear headings, a sensible structure, concrete facts, dates and authors.

Every site already has HTML. Some sites also serve a Markdown version of a page at the same address: no menus, no ads, no styling. That is easier for an agent because there is less noise in the text.

What to check: one first-level heading per page, sections named by meaning, key facts written as text rather than baked into an image.

More: Markdown versions of pages, best practices.

Two questions run through every layer.

Can this be trusted. Sources, an author and an update date are stated. The price in the markup matches the price on the page. Claims can be verified.

Does it work. Four things are measurable: bot requests in server logs, referrals from AI systems in analytics, citations, and share of mentions over a fixed set of queries. A place in an AI answer cannot be measured in advance or guaranteed.

More: How to measure AI visibility, does llms.txt work.

What is known and what is not

Confirmed

• Search engines read robots.txt, sitemap.xml and structured data: their documentation says so.
• Google and Yandex use product markup when showing price and availability.
• AI bots visit sites: their owners publish the bot names and purposes.
• Large documentation sites publish llms.txt and Markdown versions of pages.

Not proven

• That llms.txt affects search rankings or the number of citations.
• That a particular AI system reads the JSON-LD of other sites directly: it depends on how the system is built, and only its own documentation can settle it.

What it does not mean

• Markup is not required for indexing: a page will be found without it.
• The value of structured data for AI is not limited to direct reading. AI search and agents often get facts through search engines, and those do read the markup.

Each layer solves its own task. None of them promises a place in an AI answer.

How to check each item: Validating files and markup.

Some layers are assembled from data the content management system already holds: the list of pages, products, posts, authors. It makes sense to hand that work to an extension that rebuilds the files and markup on every change.

Other things cannot be automated: clear writing, accurate facts, and the decision about which robots to admit.

How it is built in one system: llms.txt and JSON-LD in Webasyst. All the extensions are listed on the developer page in the Webasyst Store. For WordPress: llms.txt in WordPress.