# An AI-ready site: llms.txt, markup and crawler access > What to do with a site so that search engines, AI models and AI agents can find and understand it - six layers from the server response to measurement Page: https://llms-full-txt.ru/en/guides/ai-ready-site/ Site index: https://llms-full-txt.ru/en/llms.txt Preparing a site for AI means a machine can fetch a page, learn whether it may read it, find the right address, understand what the page is about, and read the content itself. One file does not do that. `llms.txt` is one layer out of five. Most sites already have the other four, in varying shape. ## Five layers, one question each - What a machine learns at each layer - access rules for robots - the list of pages - a map for an AI agent - the entity, its properties and relations - the content itself The layers do not replace one another. `robots.txt` says nothing about what is useful. `llms.txt` forbids nothing. Structured data does not replace the text of a page. And the most detailed `llms.txt` will not help a site that answers with an error. ## Layer zero: the page has to arrive Before any file comes a plain requirement: the server has to answer. - The page returns `200`, not a redirect chain and not an error. - The main text is in the HTML sent by the server. If content appears only after scripts run in a browser, some automated clients will never see it. - The response is fast and stable. A fast, reliable response lowers friction for every client: a person, a search robot, an agent. - Reading the text requires no clicks, no dismissed dialogs and no scrolling. - A page has one permanent address, and links are ordinary links that can be followed. Structured data and `llms.txt` do not fix a slow or unreachable site. They describe what can already be fetched. ## 1. robots.txt: may I go `robots.txt` holds access rules. The site owner tells robots where they may go. AI companies run robots of three different classes: some collect training data, some build a search index, some come on behalf of a specific user. Blocking training while staying in AI search takes different lines in the file. What to check: the file returns `200`, the robots you need are not blocked by accident, and the decision for each class is deliberate. More: [AI crawlers and robots.txt](https://llms-full-txt.ru/en/guides/ai-crawlers/). ## 2. sitemap.xml: which addresses exist `sitemap.xml` lists the pages of a site and helps discover them. It is a complete list without judgment: the sitemap does not say what matters. `sitemap.xml` and `llms.txt` do not compete. The first answers "what exists", the second "where to start". What to check: the sitemap exists, is referenced from `robots.txt`, contains only working addresses and no pages closed to indexing. ## 3. llms.txt: where it is useful to go `llms.txt` is a map of useful material for an AI agent: the site name, a short description and lists of links with notes. The caveats matter: - it is a proposal, not an adopted standard; - search engines do not use it for ranking; - the [second edition](https://llmstxt.org/) is current: it added discovery relations and files for separate sections of a site; - `llms-full.txt` is not part of the proposal; it is an ecosystem convention. What to check: the file sits in the site root, is served as text with `200`, and its links work. More: [What is llms.txt](https://llms-full-txt.ru/en/guides/what-is-llms-txt/), [the file format](https://llms-full-txt.ru/en/specification/format/), [what is llms-full.txt](https://llms-full-txt.ru/en/guides/llms-full-txt/). ## 4. Structured data: what is on the page `llms.txt` helps a machine decide where to go. Structured data states explicitly where it has arrived: this is a product, an article or a discussion, what it is called, who wrote it, what it costs, whether it is in stock. The vocabulary for such a description is [Schema.org](https://schema.org/). The most convenient way to write it is JSON-LD: a block of data in the page code, separate from the visible text. - Route, meaning and content - llms.txt - there is a useful page about a product here - /products/widget/ - JSON-LD - this is a Product: name, SKU, price, availability - HTML / Markdown - description, specifications, instructions For search engines this is a working layer, not decoration. Google states that it uses structured data to understand the content of a page. Yandex reads product markup and, for product pages marked up as `Product` with an `Offer`, gives the data on the page the highest priority over other sources. If Schema.org can describe the kind of page and the markup is missing, the machine-readable form of the page is unfinished. More: [Structured data and JSON-LD](https://llms-full-txt.ru/en/guides/json-ld/). ## 5. HTML and Markdown: the content itself Everything else leads here. A machine needs text: clear headings, a sensible structure, concrete facts, dates and authors. Every site already has HTML. Some sites also serve a Markdown version of a page at the same address: no menus, no ads, no styling. That is easier for an agent because there is less noise in the text. What to check: one first-level heading per page, sections named by meaning, key facts written as text rather than baked into an image. More: [Markdown versions of pages](https://llms-full-txt.ru/en/guides/markdown-pages/), [best practices](https://llms-full-txt.ru/en/guides/best-practices/). ## Trust and measurement Two questions run through every layer. **Can this be trusted.** Sources, an author and an update date are stated. The price in the markup matches the price on the page. Claims can be verified. **Does it work.** Four things are measurable: bot requests in server logs, referrals from AI systems in analytics, citations, and share of mentions over a fixed set of queries. A place in an AI answer cannot be measured in advance or guaranteed. More: [How to measure AI visibility](https://llms-full-txt.ru/en/guides/ai-visibility/), [does llms.txt work](https://llms-full-txt.ru/en/guides/does-llms-txt-work/). ## What is known and what is not **Confirmed** - Search engines read `robots.txt`, `sitemap.xml` and structured data: their documentation says so. - Google and Yandex use product markup when showing price and availability. - AI bots visit sites: their owners publish the bot names and purposes. - Large documentation sites publish `llms.txt` and Markdown versions of pages. **Not proven** - That `llms.txt` affects search rankings or the number of citations. - That a particular AI system reads the JSON-LD of other sites directly: it depends on how the system is built, and only its own documentation can settle it. **What it does not mean** - Markup is not required for indexing: a page will be found without it. - The value of structured data for AI is not limited to direct reading. AI search and agents often get facts through search engines, and those do read the markup. ## A step-by-step check 1. Pages return `200`, and the main text is in the server-sent HTML. 2. `robots.txt` exists, and the decision about AI bots is deliberate. 3. `sitemap.xml` exists and is referenced from `robots.txt`. 4. Pages carry structured data of a fitting type, and the Google and Yandex validators report no errors. 5. Pages have clear headings and structure. 6. `llms.txt` sits in the site root, and its links work. 7. Price, availability and dates in the markup match what a person sees. 8. Server logs show which bots come. How to check each item: [Validating files and markup](https://llms-full-txt.ru/en/guides/validation/). ## What can be automated Some layers are assembled from data the content management system already holds: the list of pages, products, posts, authors. It makes sense to hand that work to an extension that rebuilds the files and markup on every change. Other things cannot be automated: clear writing, accurate facts, and the decision about which robots to admit. How it is built in one system: [llms.txt and JSON-LD in Webasyst](https://llms-full-txt.ru/en/guides/webasyst/). All the extensions are listed on the developer page in the Webasyst Store. For WordPress: [llms.txt in WordPress](https://llms-full-txt.ru/en/guides/wordpress/).