AI crawlers guide: who to allow, who to block

Update, October 3, 2026. This post is kept as published on February 15, and its table is inaccurate. GPTBot and ClaudeBot cover training only; search and user requests are served by other bots. No bot owner states that it reads other sites’ llms.txt, so the file should not be expected to raise visibility in AI answers. The table checked against the owners’ documentation: AI crawlers and robots.txt.

AI Crawlers in 2026

The AI crawler landscape has changed significantly. Today, more than 15 different bots regularly scan websites for model training and inference.

Main AI Crawlers

Bot	Company	Purpose
GPTBot	OpenAI	Training + inference
ClaudeBot	Anthropic	Training + inference
Google-Extended	Google	Training Gemini
PerplexityBot	Perplexity	Search index

Recommendations

For documentation sites, allowing all AI crawlers is recommended — it increases your content visibility in AI responses. The llms.txt file additionally helps bots find structured information.