# llmstxt-architect: generating llms.txt with an LLM

> llmstxt-architect is a Python utility that crawls a site and writes llms.txt link notes with a language model: running it, options, output, and why to treat it as frozen

Page: https://llms-full-txt.ru/en/tools/llmstxt-architect/
Site index: https://llms-full-txt.ru/en/llms.txt

`llmstxt-architect` crawls the pages of a site, hands the text of each to a language model and gets a note back: what the page is about and when it is worth reading. The notes are assembled into `llms.txt`.

Its strength is meaningful notes instead of copied page descriptions.

Its weakness is that [the project](https://github.com/rlancemartin/llmstxt_architect) has not been updated since March 2025. It needs access to a language model: a paid key or a local model.

no releases since March 2025
- Author: rlancemartin
- Package: llmstxt-architect on PyPI
- Version: 0.6.1, March 29, 2025
- License: MIT
- Checked: October 3, 2026

## Running

Without installation, through `uvx`:

```bash
uvx --from llmstxt-architect llmstxt-architect \
  --urls https://example.com/docs \
  --max-depth 1 \
  --llm-name <model-name> \
  --llm-provider anthropic \
  --project-dir out
```

Or installed into an environment:

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install llmstxt-architect
```

The model provider's key is passed as an environment variable, for example `ANTHROPIC_API_KEY` or `OPENAI_API_KEY`. A local model through Ollama additionally needs `langchain-ollama`.

The model name in the project's examples has aged together with the project. Put in a current model name from your provider.

## Options

| Option | Default | What it does |
|---|---|---|
| `--urls` | - | addresses to start crawling from |
| `--existing-llms-file` | - | take addresses from an existing `llms.txt`: a local path or an address |
| `--update-descriptions-only` | off | refresh only the notes, keeping structure and order |
| `--max-depth` | 5 | crawl depth |
| `--llm-name` | an old model | the model name |
| `--llm-provider` | `anthropic` | the model provider |
| `--project-dir` | `llms_txt` | the folder with results |
| `--output-dir` | `summaries` | the folder with per-page notes |
| `--output-file` | `llms.txt` | the name of the final file |
| `--summary-prompt` | built in | your own instruction for the model |
| `--blacklist-file` | none | a file with addresses to skip |
| `--extractor` | `default` | how text is extracted: `default` or `bs4` |

## Refreshing notes in an existing file

A useful mode: the structure is already thought through, and the notes are weak.

```bash
llmstxt-architect \
  --existing-llms-file ./llms.txt \
  --update-descriptions-only \
  --llm-name <model-name> \
  --llm-provider anthropic \
  --project-dir out
```

Without `--update-descriptions-only` the structure of the source file is not preserved.

## What comes out

```text
out/
  llms.txt                       the final file
  summaries/
    <address>.txt                a note for each page
    summarized_urls.json         progress marks
```

A line of the final file looks like `[Title](address): a note from the model`. When a run is interrupted, repeating the same command resumes where it stopped.

The utility does not create `llms-full.txt`.

## Calling it from code

```python
import asyncio
from llmstxt_architect.main import generate_llms_txt

asyncio.run(generate_llms_txt(
    urls=["https://example.com/docs"],
    max_depth=1,
    llm_name="<model-name>",
    llm_provider="anthropic",
    project_dir="out",
))
```

## Limits

- **A language model is required.** Every page is a separate call, and on a large site that costs money.
- **No sections.** The output is one list of links.
- **Notes have to be read.** A model can describe a page inaccurately.
- **The project is not developing.** Nobody will fix bugs, and dependencies age.

## When it fits

The utility is justified for one-off work: assemble a draft of notes for large documentation, then fix it by hand. For continuous refreshing, take a tool that runs during the site build.

The same work can be done without the utility by handing a list of pages to an AI system: [a prompt for generation](https://llms-full-txt.ru/en/prompts/generate-llms-txt/).

Other options: [Generators and tools](https://llms-full-txt.ru/en/tools/generators/).