# GPTBot, ClaudeBot and robots.txt: managing AI crawlers

> AI bots of OpenAI, Anthropic, Google, Perplexity, Apple, Meta and Yandex: three classes (training, search, user-triggered), which obey robots.txt, and a configuration template

Page: https://llms-full-txt.ru/en/guides/ai-crawlers/
Site index: https://llms-full-txt.ru/en/llms.txt

Bots of AI companies fall into three classes: training, search and user request. The rules for them differ, and blocking one class does not block another.

Access is controlled by `robots.txt`. `llms.txt` forbids nothing: it only shows where the important material is.

- Checked: October 3, 2026
- Source: pages of the bot owners
- Owners in the table: 12

The lists change. Before configuring anything, check the owner's page: the links are at the end.

## Three classes of bots

- Who comes to a site and what stops them
- Training
- on its own, on a schedule
- Training bot
- GPTBot, ClaudeBot
- robots.txt
- obeyed
- Dataset
- may be used to train models
- Search
- on its own, on a schedule
- Search bot
- OAI-SearchBot, PerplexityBot
- robots.txt
- obeyed
- Index
- the page in an answer with a link
- User request
- because a person asked
- A person
- gives a link or asks a question
- Assistant
- ChatGPT-User, Claude-User
- robots.txt
- often does not apply

## All bots by owner

| Owner | Training | Search | User request |
|---|---|---|---|
| OpenAI | `GPTBot` - content may be used to train models | `OAI-SearchBot` - shown in ChatGPT search; a rule applies in about a day | `ChatGPT-User` - rules may not apply |
| Anthropic | `ClaudeBot` - may be used for training; honors Crawl-delay | `Claude-SearchBot` - search result quality | `Claude-User` - a disallow blocks the fetch |
| Google | `Google-Extended` - a control token, no separate bot: Gemini training | `Googlebot` - also governs AI Overviews and AI Mode | `Google-Agent` - generally ignores; same for Google-NotebookLM |
| Perplexity | none | `PerplexityBot` - links in results; not used to train foundation models | `Perplexity-User` - generally ignores |
| Apple | `Applebot-Extended` - a control token: opt out of Apple model training | `Applebot` - Spotlight, Siri, Safari | none |
| Amazon | `Amazonbot` - the noarchive meta tag opts out of training | `Amzn-SearchBot` - search and Alexa; not used for training | `Amzn-User` - may not follow every rule |
| Meta | `meta-externalagent` - model training and product improvement | `meta-webindexer` - Meta AI search quality | `meta-externalfetcher` - may bypass rules |
| Mistral | `MistralAI-Training` - dataset collection | `MistralAI-Index` - indexing only | `MistralAI-User` - the owner does not say |
| Common Crawl | `CCBot` - an open web archive, also used for training | none | none |
| DuckDuckGo | none | `DuckAssistBot` - AI answers; a rule applies within 72 hours | none |
| Microsoft | none | `bingbot` - the regular search bot, no separate AI bot | none |
| Яндекс | `YandexAdditional` - a control token: use in Yandex AI answers in Search | `YandexBot` - the main indexing robot | none |

## Do not confuse

- Three names, three separate settings
- OpenAI
- GPTBot
- training
- OAI-SearchBot
- ChatGPT search
- ChatGPT-User
- user request
- Blocking GPTBot does not close a site to ChatGPT. According to OpenAI, each setting is independent of the others.
- Anthropic
- ClaudeBot
- training
- Claude-SearchBot
- search
- Claude-User
- user request
- The names anthropic-ai and Claude-Web are not on the Anthropic page. There is no need to block them separately.
- Google
- Google-Extended
- Gemini training
- Googlebot
- Search and AI Overviews
- Blocking Google-Extended does not remove a site from AI Overviews. Those are governed by Googlebot rules and by nosnippet, data-nosnippet, max-snippet and noindex.

## Yandex

Yandex has a separate control token for AI answers:

```text title="robots.txt"
User-agent: YandexAdditional
Disallow: /
```

It forbids showing page content in Yandex AI answers in Search. This robot makes no indexing requests. The general `User-agent: *` rule does not apply to it: the name has to be given explicitly. According to Yandex, a change takes effect in 2-14 days.

## Ready-made settings

**No training, search allowed**

```text title="robots.txt"
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: MistralAI-Training
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml
```

**Documentation only**

```text title="robots.txt"
User-agent: GPTBot
Allow: /docs/
Disallow: /
```

**No Yandex AI answers**

```text title="robots.txt"
User-agent: YandexAdditional
Disallow: /
```

## What robots.txt does not have yet

**Confirmed**

- Amazon treats the `noarchive` meta tag as an opt-out of training.
- Bing uses `NOCACHE` and `NOARCHIVE` to control use in chat and training.
- [Cloudflare documents](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) the `Content-Signal` line and adds it to its managed `robots.txt` by default.

**Not proven**

- `DisallowAITraining`: a draft by Microsoft employees expired and was not adopted by the IETF. No bot owner claims support.
- The `noai` and `noimageai` meta tags appeared at DeviantArt in 2022. Bot owners do not confirm support.

**What it does not mean**

- `Content-Signal` is an initiative of one company, not a standard.
- The IETF [AIPREF](https://datatracker.ietf.org/wg/aipref/about/) working group is preparing a vocabulary of preferences. As of October 3, 2026 these are drafts; there is no RFC.

## How to verify that a bot is real

A bot name in the `User-Agent` header is easy to fake. Owners publish address lists or offer a DNS check.

| Owner | Where the list is |
|---|---|
| OpenAI | `openai.com/gptbot.json`, `searchbot.json`, `chatgpt-user.json` |
| Anthropic | `claude.com/crawling/bots.json` |
| Perplexity | `perplexity.com/perplexitybot.json`, `perplexity-user.json` |
| Apple | reverse DNS `*.applebot.apple.com` |
| Common Crawl | reverse DNS `*.crawl.commoncrawl.org` |
| DuckDuckGo | `duckduckgo.com/duckassistbot.json` |
| Yandex | reverse DNS in `yandex.ru`, `yandex.net`, `yandex.com` and a forward check |

## Do these bots read llms.txt

**Confirmed**

- OpenAI, Anthropic, Perplexity and Yandex publish `llms.txt` for their own documentation.
- Google says it plainly: Search does not use such files.

**Not proven**

- That a bot of at least one owner reads someone else's `/llms.txt`. No owner claims it.

**What it does not mean**

- Publishing its own file does not oblige a company's bots to read yours.

More in [llms.txt status](https://llms-full-txt.ru/en/guides/standard-status/).

## llms.txt and robots.txt

- Two files, two jobs
- robots.txt
- where not to go
- Controls access
- Addressed to all bots
- Its own syntax
- A standard: RFC 9309
- llms.txt
- where the important material is
- Does not control access
- Addressed to agents and models
- Markdown
- A proposal by one author

The files complement each other: the first restricts, the second guides.

## What we do not know

- `Bytespider` (ByteDance) has no official page describing it. Its purpose and its treatment of `robots.txt` are not confirmed.
- No official pages about bots were found for Cohere and xAI.
- The owner's page does not give full `User-Agent` strings of Anthropic bots.