# AI crawlers guide: who to allow, who to block

> Complete guide to AI crawlers in 2026 — GPTBot, ClaudeBot, Google-Extended. How to configure robots.txt for AI bots.

Page: https://llms-full-txt.ru/en/blog/ai-crawlers-guide/
Site index: https://llms-full-txt.ru/en/llms.txt

Published: 2026-02-15

> **Update, October 3, 2026.** This post is kept as published on February 15, and its table is inaccurate. `GPTBot` and `ClaudeBot` cover training only; search and user requests are served by other bots. No bot owner states that it reads other sites' llms.txt, so the file should not be expected to raise visibility in AI answers. The table checked against the owners' documentation: [AI crawlers and robots.txt](https://llms-full-txt.ru/en/guides/ai-crawlers/).

## AI Crawlers in 2026

The AI crawler landscape has changed significantly. Today, more than 15 different bots regularly scan websites for model training and inference.

## Main AI Crawlers

| Bot | Company | Purpose |
|-----|---------|---------|
| GPTBot | OpenAI | Training + inference |
| ClaudeBot | Anthropic | Training + inference |
| Google-Extended | Google | Training Gemini |
| PerplexityBot | Perplexity | Search index |

## Recommendations

For documentation sites, allowing all AI crawlers is recommended — it increases your content visibility in AI responses. The llms.txt file additionally helps bots find structured information.