llmstxt-architect

llmstxt-architect - a Python utility that crawls a site and writes the link notes of llms.txt with a language model - running it, options, what comes out, and why the project should be treated as frozen

no releases since March 2025

Author: rlancemartin
Package: llmstxt-architect on PyPI
Version: 0.6.1, March 29, 2025
License: MIT
Checked: October 3, 2026

Running

Without installation, through uvx:

uvx --from llmstxt-architect llmstxt-architect \
  --urls https://example.com/docs \
  --max-depth 1 \
  --llm-name <model-name> \
  --llm-provider anthropic \
  --project-dir out

Or installed into an environment:

python3 -m venv .venv
source .venv/bin/activate
pip install llmstxt-architect

The model provider’s key is passed as an environment variable, for example ANTHROPIC_API_KEY or OPENAI_API_KEY. A local model through Ollama additionally needs langchain-ollama.

The model name in the project’s examples has aged together with the project. Put in a current model name from your provider.

Options

Option	Default	What it does
`--urls`	-	addresses to start crawling from
`--existing-llms-file`	-	take addresses from an existing `llms.txt`: a local path or an address
`--update-descriptions-only`	off	refresh only the notes, keeping structure and order
`--max-depth`	5	crawl depth
`--llm-name`	an old model	the model name
`--llm-provider`	`anthropic`	the model provider
`--project-dir`	`llms_txt`	the folder with results
`--output-dir`	`summaries`	the folder with per-page notes
`--output-file`	`llms.txt`	the name of the final file
`--summary-prompt`	built in	your own instruction for the model
`--blacklist-file`	none	a file with addresses to skip
`--extractor`	`default`	how text is extracted: `default` or `bs4`

Refreshing notes in an existing file

A useful mode: the structure is already thought through, and the notes are weak.

llmstxt-architect \
  --existing-llms-file ./llms.txt \
  --update-descriptions-only \
  --llm-name <model-name> \
  --llm-provider anthropic \
  --project-dir out

Without --update-descriptions-only the structure of the source file is not preserved.

What comes out

out/
  llms.txt                       the final file
  summaries/
    <address>.txt                a note for each page
    summarized_urls.json         progress marks

A line of the final file looks like [Title](address): a note from the model. When a run is interrupted, repeating the same command resumes where it stopped.

The utility does not create llms-full.txt.

Calling it from code

import asyncio
from llmstxt_architect.main import generate_llms_txt

asyncio.run(generate_llms_txt(
    urls=["https://example.com/docs"],
    max_depth=1,
    llm_name="<model-name>",
    llm_provider="anthropic",
    project_dir="out",
))

Limits

A language model is required. Every page is a separate call, and on a large site that costs money.
No sections. The output is one list of links.
Notes have to be read. A model can describe a page inaccurately.
The project is not developing. Nobody will fix bugs, and dependencies age.

When it fits

The utility is justified for one-off work: assemble a draft of notes for large documentation, then fix it by hand. For continuous refreshing, take a tool that runs during the site build.

The same work can be done without the utility by handing a list of pages to an AI system: a prompt for generation.

Other options: Generators and tools.