GPTBot, ClaudeBot, PerplexityBot: Who Is Crawling You and What to Allow

These are the user-agent tokens the major AI engines use to fetch your pages, and the practical rule is short. Allow the crawlers that fetch pages to build live answers if you want to appear in those answers. Decide separately about the crawlers that gather training data, because blocking those costs you nothing in visibility. And never confuse any of them with Googlebot, which is the crawler that actually gates Google's AI Overviews. ClarAI's Tech SEO Audit checks seven of these tokens by name and labels each by role, the retrieval agents OAI-SearchBot, PerplexityBot, Claude-SearchBot and Claude-User, and the training crawlers GPTBot and ClaudeBot plus the Google-Extended policy token, with a dedicated Googlebot check that outranks them all. This post explains what each one is, the one distinction that decides whether blocking it hurts, and how to read your own robots.txt in a couple of minutes.

The tokens ClarAI checks, by role, plus the one it ranks above them

• GPTBot, from OpenAI. Per OpenAI's published crawler documentation, GPTBot crawls content that may be used to train its generative models. It is a training crawler.
• OAI-SearchBot, also from OpenAI. Per OpenAI's documentation, this one surfaces and links to sites in ChatGPT search results and is not used to train the foundation models. It is a retrieval crawler, and it is the OpenAI token that matters for appearing in ChatGPT's answers.
• ClaudeBot, from Anthropic. Per Anthropic's crawler documentation, ClaudeBot collects web content that may contribute to model training: it is the training crawler. The Anthropic tokens that decide answer visibility are Claude-SearchBot, which Anthropic says improves search result quality and whose blocking may reduce your visibility in Claude's search responses, and Claude-User, which fetches on a user's behalf and whose blocking may reduce visibility for user-directed web search. Blocking ClaudeBot alone does not remove you from Claude's answers.
• PerplexityBot, from Perplexity. Per Perplexity's documentation it surfaces sites in Perplexity results, respects robots.txt and is not used for training. It is a retrieval crawler. Perplexity-User is the user-triggered fetcher and, per Perplexity, generally ignores robots.txt because a user requested the page, so a robots rule is not the lever for it.
• Google-Extended, from Google. This one is not a crawler that fetches your pages at all; it is a robots.txt token that controls how already-crawled content may be used. More on why that matters below.
• Googlebot, which ClarAI checks separately and ranks above all of the above, because it is the crawler that actually gates Google's AI Overviews and AI Mode.

Updated 1 October 2026: the vendors' lists have moved since this post was published. Google added a Google-Agent token for user-triggered fetches, which generally ignore robots.txt. OpenAI lists a fourth bot, OAI-AdsBot, which fetches advertising landing pages and has no bearing on answers. Apple's page (dated 4 September 2026) says Applebot-Extended does not crawl and only lets publishers opt out of their content being used to train Apple's foundation models; pages that disallow it can still appear in search results. To keep a page's text out of the AI answers Siri and Search give to broad questions, Apple points to the nosnippet meta tag instead. ClarAI's audit now also reads Bingbot, Applebot and the user-triggered ChatGPT-User and Perplexity-User fetchers, and applies robots.txt rules per page path, not only to the whole site. Our learn page on AI crawlers keeps the current roster, by role. Corrected 13 October 2026: this note first said Applebot-Extended also governs answers in Siri and Search, so it was not a pure training opt-out. That misread Apple's page, which says it covers training only and that its rules are not considered in ranking. The same day we corrected how this post describes Google-Extended: besides Gemini training and grounding in the Gemini app and Vertex AI, Search Console's help says it is the control for limiting training of the models behind Search's generative AI features, though it still does not decide whether you appear in them.

Two kinds of crawler, two different consequences

The single distinction that makes this manageable: some AI crawlers gather content to train models, and others fetch pages live to compose an answer a user is reading right now. Blocking a training crawler protects your content from model training at no visibility cost. Blocking a retrieval crawler removes you from that engine's answers. The two carry completely different consequences, so before you add any Disallow rule, decide which kind of crawler you are blocking.

OpenAI's own split is the clearest worked example. GPTBot is the training crawler; OAI-SearchBot is the retrieval crawler that surfaces sites inside ChatGPT search. Block GPTBot and you keep your content out of training while staying eligible to appear in ChatGPT's answers. Block OAI-SearchBot and you remove yourself from those answers. They look almost identical in a robots.txt file and do opposite jobs, which is exactly why reading the vendor documentation before blocking is worth the two minutes.

Google-Extended is not what it sounds like

Google-Extended is the token most often misunderstood, and getting it wrong is costly. Per Google's crawler documentation, Google-Extended governs whether your content is used to train Gemini models and to ground answers in the Gemini app and in Grounding with Google Search on Vertex AI, and Search Console's help adds that it is the control for limiting training of the models behind Search's generative AI features. It does not control whether you appear in Google's AI Overviews or AI Mode. Those features draw their answers from the ordinary Search index, so the crawler that gates them is standard Googlebot.

“To be eligible to be shown in generative AI features on Google Search, a page must be indexed and eligible to be shown in Google Search with a snippet.”

Read that against what Google-Extended does and the two stop being interchangeable. Being in Google's AI answers runs through Googlebot, indexing and snippet eligibility. Google-Extended sits entirely outside that path. ClarAI's audit reflects this directly: a Googlebot disallow is raised as a hard failure, because it removes you from Google Search and every Google AI feature at once, while a Google-Extended-only block is deliberately downgraded to a warning rather than a failure, because it does not cost you AI Overviews or AI Mode visibility. The most damaging robots.txt line for AI visibility is not any of the AI tokens: it is a Disallow that applies to Googlebot.

How to read your robots.txt in two minutes

• Open yourdomain/robots.txt in a browser. It is plain text and usually short enough to read top to bottom.
• Find the Googlebot group and scan for any Disallow. A Disallow: / under Googlebot, or a global User-agent: * Disallow: /, is the line that removes you from Google Search and Google AI answers. This is the one that matters most.
• Find the Google-Extended group. A Disallow here opts you out of Gemini training and of grounding in the Gemini app. It is not costing you AI Overviews or AI Mode visibility, so set it however you have genuinely decided about training and the Gemini app.
• Check the retrieval crawlers you care about, OAI-SearchBot for ChatGPT search and PerplexityBot for Perplexity. A Disallow on one of these removes you from that engine's live answers.
• Remember RFC 9309: a crawler with its own named group ignores the wildcard User-agent: * group, so a bot-specific block overrides the general one. If you want a per-bot policy, give the bot its own group.

What ClarAI's check does

ClarAI checks this in two places. Inside the app, the Tech SEO Audit resolves the AI-crawler allowlist by role, retrieval agents separately from training crawlers, plus the dedicated Googlebot check, flags an implicit allow (a bot with no Disallow but no explicit Allow either) as a nudge to add an explicit User-agent block, and ships each finding with a copy-ready robots.txt fix. The free AI Readiness Grader at /grade weights this heavily on purpose: Googlebot access is about 22 percent of a fully measured grade (20 of the 91 points across the seven scored checks) and retrieval-crawler access about 16 percent (15 of 91), because these are the binary gates on inclusion; a blocked training crawler is reported on its own unscored row. If only Google-Extended or another training token is blocked, the grader says so plainly and does not collapse your grade the way a Googlebot block does. No signup, and no AI cost to run it. Updated 19 October 2026: the grader's Markdown-for-agents row stopped scoring and is now shown for information only, so these shares are now out of 91 points across seven scored checks.

The honest limit worth stating: being crawlable makes you eligible, it does not guarantee you get cited. Passing the crawler checks is the floor, not the finish line, which is why answer-and-citation monitoring exists alongside them.

Quick answers to the common confusions

• Does blocking GPTBot remove me from ChatGPT's answers? Not by itself. GPTBot is OpenAI's training crawler; the token that governs appearing in ChatGPT search is OAI-SearchBot. Check OpenAI's crawler documentation and block the one that matches your intent.
• Do I have to allow Google-Extended to appear in AI Overviews? No. Google's AI features depend on Googlebot access plus indexing and snippet eligibility. Allowing Google-Extended is not required for them.
• Can I opt out of AI training and still keep AI-answer visibility? Yes, with one Google caveat. Block the training crawlers and leave the retrieval crawlers and Googlebot allowed. If you block Google-Extended, Google also stops using your pages to ground answers in the Gemini app, while Google Search, AI Overviews and AI Mode are unaffected.
• What is the single most damaging robots.txt line for AI visibility? A Disallow: / that applies to Googlebot, whether under User-agent: Googlebot or a global User-agent: *. It removes you from Google Search and every Google AI answer at once.