GPTBot, ClaudeBot, PerplexityBot: Who Is Crawling You and What to Allow

These are the user-agent tokens the major AI engines use to fetch your pages, and the practical rule is short. Allow the crawlers that fetch pages to build live answers if you want to appear in those answers. Decide separately about the crawlers that gather training data, because blocking those costs you nothing in visibility. And never confuse any of them with Googlebot, which is the crawler that actually gates Google's AI Overviews. ClarAI's Tech SEO Audit checks five of these tokens by name, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and OAI-SearchBot, plus a dedicated Googlebot check that outranks them all. This post explains what each one is, the one distinction that decides whether blocking it hurts, and how to read your own robots.txt in a couple of minutes.

The five tokens ClarAI checks, plus the one it ranks above them

• GPTBot, from OpenAI. Per OpenAI's published crawler documentation, GPTBot crawls content that may be used to train its generative models. It is a training crawler.
• OAI-SearchBot, also from OpenAI. Per OpenAI's documentation, this one surfaces and links to sites in ChatGPT search results and is not used to train the foundation models. It is a retrieval crawler, and it is the OpenAI token that matters for appearing in ChatGPT's answers.
• ClaudeBot, from Anthropic, and PerplexityBot, from Perplexity. Each engine publishes its own crawler documentation, and that documentation is the authority on exactly what the crawler does. Check the vendor's own page before you decide to block either.
• Google-Extended, from Google. This one is not a crawler that fetches your pages at all; it is a robots.txt token that controls how already-crawled content may be used. More on why that matters below.
• Googlebot, which ClarAI checks separately and ranks above all of the above, because it is the crawler that actually gates Google's AI Overviews and AI Mode.

Two kinds of crawler, two different consequences

The single distinction that makes this manageable: some AI crawlers gather content to train models, and others fetch pages live to compose an answer a user is reading right now. Blocking a training crawler protects your content from model training at no visibility cost. Blocking a retrieval crawler removes you from that engine's answers. The two carry completely different consequences, so before you add any Disallow rule, decide which kind of crawler you are blocking.

OpenAI's own split is the clearest worked example. GPTBot is the training crawler; OAI-SearchBot is the retrieval crawler that surfaces sites inside ChatGPT search. Block GPTBot and you keep your content out of training while staying eligible to appear in ChatGPT's answers. Block OAI-SearchBot and you remove yourself from those answers. They look almost identical in a robots.txt file and do opposite jobs, which is exactly why reading the vendor documentation before blocking is worth the two minutes.

Google-Extended is not what it sounds like

Google-Extended is the token most often misunderstood, and getting it wrong is costly. Per Google's published guidance, Google-Extended governs whether your content is used to train and ground Gemini in Google's other products. It does not control whether you appear in Google's AI Overviews or AI Mode. Those features draw their answers from the ordinary Search index, so the crawler that gates them is standard Googlebot.

“To be eligible to be shown in generative AI features on Google Search, a page must be indexed and eligible to be shown in Google Search with a snippet.”

Read that against what Google-Extended does and the two stop being interchangeable. Being in Google's AI answers runs through Googlebot, indexing and snippet eligibility. Google-Extended sits entirely outside that path. ClarAI's audit reflects this directly: a Googlebot disallow is raised as a hard failure, because it removes you from Google Search and every Google AI feature at once, while a Google-Extended-only block is deliberately downgraded to a warning rather than a failure, because it does not cost you Google AI visibility. The most damaging robots.txt line for AI visibility is not any of the AI tokens: it is a Disallow that applies to Googlebot.

How to read your robots.txt in two minutes

• Open yourdomain/robots.txt in a browser. It is plain text and usually short enough to read top to bottom.
• Find the Googlebot group and scan for any Disallow. A Disallow: / under Googlebot, or a global User-agent: * Disallow: /, is the line that removes you from Google Search and Google AI answers. This is the one that matters most.
• Find the Google-Extended group. A Disallow here is a training opt-out and nothing more. It is not costing you AI Overviews visibility, so set it however you have genuinely decided about training.
• Check the retrieval crawlers you care about, OAI-SearchBot for ChatGPT search and PerplexityBot for Perplexity. A Disallow on one of these removes you from that engine's live answers.
• Remember RFC 9309: a crawler with its own named group ignores the wildcard User-agent: * group, so a bot-specific block overrides the general one. If you want a per-bot policy, give the bot its own group.

What ClarAI's check does

ClarAI checks this in two places. Inside the app, the Tech SEO Audit resolves the five-token AI-crawler allowlist plus the dedicated Googlebot check, flags an implicit allow (a bot with no Disallow but no explicit Allow either) as a nudge to add an explicit User-agent block, and ships each finding with a copy-ready robots.txt fix. The free AI Readiness Grader at /grade weights this heavily on purpose: Googlebot access is worth 20 of the report's 100 points and AI-crawler access another 15, because these are the binary gates on inclusion. If only Google-Extended is blocked, the grader says so plainly and does not collapse your grade the way a Googlebot block does. No signup, and no AI cost to run it.

The honest limit worth stating: being crawlable makes you eligible, it does not guarantee you get cited. Passing the crawler checks is the floor, not the finish line, which is why answer-and-citation monitoring exists alongside them.

Quick answers to the common confusions

• Does blocking GPTBot remove me from ChatGPT's answers? Not by itself. GPTBot is OpenAI's training crawler; the token that governs appearing in ChatGPT search is OAI-SearchBot. Check OpenAI's crawler documentation and block the one that matches your intent.
• Do I have to allow Google-Extended to appear in AI Overviews? No. Google's AI features depend on Googlebot access plus indexing and snippet eligibility. Allowing Google-Extended is not required for them.
• Can I opt out of AI training and still keep AI-answer visibility? Yes. Block the training crawlers and leave the retrieval crawlers and Googlebot allowed. The two decisions are independent.
• What is the single most damaging robots.txt line for AI visibility? A Disallow: / that applies to Googlebot, whether under User-agent: Googlebot or a global User-agent: *. It removes you from Google Search and every Google AI answer at once.