Skip to content
LLMs Full Text
Select theme

AI crawlers and robots.txt

Bots of OpenAI, Anthropic, Google, Perplexity, Apple, Amazon, Meta, Mistral and Yandex from their owners' documentation - three classes (training, search, user-triggered), what obeys robots.txt, and a configuration template

Checked
October 3, 2026
Source
pages of the bot owners
Owners in the table
12

The lists change. Before configuring anything, check the owner’s page: the links are at the end.

Three classes of bots

Section titled “Three classes of bots”
Who comes to a site and what stops them

Trainingon its own, on a schedule

  1. Training botGPTBot, ClaudeBot
  2. robots.txtobeyed
  3. Datasetmay be used to train models

Searchon its own, on a schedule

  1. Search botOAI-SearchBot, PerplexityBot
  2. robots.txtobeyed
  3. Indexthe page in an answer with a link

User requestbecause a person asked

  1. A persongives a link or asks a question
  2. AssistantChatGPT-User, Claude-User
  3. robots.txtoften does not apply

The third class opens a page not on its own but because a person asked. That is why owners say plainly that robots.txt rules may not apply to such requests.

All bots by owner

Section titled “All bots by owner”
AI bots by owner and class
OwnerTrainingcollects pages to train modelsSearchindexes pages to cite them in answersUser requestopens a page because a person asked
OpenAITrainingGPTBotobeys robots.txtcontent may be used to train modelsSearchOAI-SearchBotobeys robots.txtshown in ChatGPT search; a rule applies in about a dayUser requestChatGPT-Usermay ignore robots.txtrules may not apply
AnthropicTrainingClaudeBotobeys robots.txtmay be used for training; honors Crawl-delaySearchClaude-SearchBotobeys robots.txtsearch result qualityUser requestClaude-Userobeys robots.txta disallow blocks the fetch
GoogleTrainingGoogle-Extendedrobots.txt tokena control token, no separate bot: Gemini trainingSearchGooglebotobeys robots.txtalso governs AI Overviews and AI ModeUser requestGoogle-Agentmay ignore robots.txtgenerally ignores; same for Google-NotebookLM
PerplexityTrainingno bot of this classSearchPerplexityBotobeys robots.txtlinks in results; not used to train foundation modelsUser requestPerplexity-Usermay ignore robots.txtgenerally ignores
AppleTrainingApplebot-Extendedrobots.txt tokena control token: opt out of Apple model trainingSearchApplebotobeys robots.txtSpotlight, Siri, SafariUser requestno bot of this class
AmazonTrainingAmazonbotobeys robots.txtthe noarchive meta tag opts out of trainingSearchAmzn-SearchBotobeys robots.txtsearch and Alexa; not used for trainingUser requestAmzn-Usermay ignore robots.txtmay not follow every rule
MetaTrainingmeta-externalagentobeys robots.txtmodel training and product improvementSearchmeta-webindexerobeys robots.txtMeta AI search qualityUser requestmeta-externalfetchermay ignore robots.txtmay bypass rules
MistralTrainingMistralAI-Trainingobeys robots.txtdataset collectionSearchMistralAI-Indexobeys robots.txtindexing onlyUser requestMistralAI-Usernot statedthe owner does not say
Common CrawlTrainingCCBotobeys robots.txtan open web archive, also used for trainingSearchno bot of this classUser requestno bot of this class
DuckDuckGoTrainingno bot of this classSearchDuckAssistBotobeys robots.txtAI answers; a rule applies within 72 hoursUser requestno bot of this class
MicrosoftTrainingno bot of this classSearchbingbotobeys robots.txtthe regular search bot, no separate AI botUser requestno bot of this class
ЯндексTrainingYandexAdditionalrobots.txt tokena control token: use in Yandex AI answers in SearchSearchYandexBotobeys robots.txtthe main indexing robotUser requestno bot of this class

Do not confuse

Section titled “Do not confuse”

Three names, three separate settings

OpenAI

  • GPTBottraining
  • OAI-SearchBotChatGPT search
  • ChatGPT-Useruser request

Blocking GPTBot does not close a site to ChatGPT. According to OpenAI, each setting is independent of the others.

Anthropic

  • ClaudeBottraining
  • Claude-SearchBotsearch
  • Claude-Useruser request

The names anthropic-ai and Claude-Web are not on the Anthropic page. There is no need to block them separately.

Google

  • Google-ExtendedGemini training
  • GooglebotSearch and AI Overviews

Blocking Google-Extended does not remove a site from AI Overviews. Those are governed by Googlebot rules and by nosnippet, data-nosnippet, max-snippet and noindex.

Yandex

Section titled “Yandex”

Yandex has a separate control token for AI answers:

robots.txt
User-agent: YandexAdditional
Disallow: /

It forbids showing page content in Yandex AI answers in Search. This robot makes no indexing requests. The general User-agent: * rule does not apply to it: the name has to be given explicitly. According to Yandex, a change takes effect in 2-14 days.

Ready-made settings

Section titled “Ready-made settings”
robots.txt
User-agent: GPTBot
Disallow: /


User-agent: ClaudeBot
Disallow: /


User-agent: Google-Extended
Disallow: /


User-agent: Applebot-Extended
Disallow: /


User-agent: meta-externalagent
Disallow: /


User-agent: MistralAI-Training
Disallow: /


User-agent: CCBot
Disallow: /


User-agent: OAI-SearchBot
Allow: /


User-agent: Claude-SearchBot
Allow: /


User-agent: PerplexityBot
Allow: /


Sitemap: https://example.com/sitemap.xml

What robots.txt does not have yet

Section titled “What robots.txt does not have yet”

Signals for AI: what actually works

Confirmed

  • Amazon treats the noarchive meta tag as an opt-out of training.
  • Bing uses NOCACHE and NOARCHIVE to control use in chat and training.
  • Cloudflare documents the Content-Signal line and adds it to its managed robots.txt by default.

Not proven

  • DisallowAITraining: a draft by Microsoft employees expired and was not adopted by the IETF. No bot owner claims support.
  • The noai and noimageai meta tags appeared at DeviantArt in 2022. Bot owners do not confirm support.

What it does not mean

  • Content-Signal is an initiative of one company, not a standard.
  • The IETF AIPREF working group is preparing a vocabulary of preferences. As of October 3, 2026 these are drafts; there is no RFC.

How to verify that a bot is real

Section titled “How to verify that a bot is real”

A bot name in the User-Agent header is easy to fake. Owners publish address lists or offer a DNS check.

Owner Where the list is
OpenAI openai.com/gptbot.json, searchbot.json, chatgpt-user.json
Anthropic claude.com/crawling/bots.json
Perplexity perplexity.com/perplexitybot.json, perplexity-user.json
Apple reverse DNS *.applebot.apple.com
Common Crawl reverse DNS *.crawl.commoncrawl.org
DuckDuckGo duckduckgo.com/duckassistbot.json
Yandex reverse DNS in yandex.ru, yandex.net, yandex.com and a forward check

Do these bots read llms.txt

Section titled “Do these bots read llms.txt”

What is known and what is not

Confirmed

  • OpenAI, Anthropic, Perplexity and Yandex publish llms.txt for their own documentation.
  • Google says it plainly: Search does not use such files.

Not proven

  • That a bot of at least one owner reads someone else’s /llms.txt. No owner claims it.

What it does not mean

  • Publishing its own file does not oblige a company’s bots to read yours.

More in llms.txt status.

llms.txt and robots.txt

Section titled “llms.txt and robots.txt”
Two files, two jobs

robots.txt

where not to go

  • Controls access
  • Addressed to all bots
  • Its own syntax
  • A standard: RFC 9309

llms.txt

where the important material is

  • Does not control access
  • Addressed to agents and models
  • Markdown
  • A proposal by one author

The files complement each other: the first restricts, the second guides.

What we do not know

Section titled “What we do not know”
  • Bytespider (ByteDance) has no official page describing it. Its purpose and its treatment of robots.txt are not confirmed.
  • No official pages about bots were found for Cohere and xAI.
  • The owner’s page does not give full User-Agent strings of Anthropic bots.