Google-Extended Does Not Do What You Think It Does

Blocking Google-Extended in your robots.txt does not remove your brand from Google's AI Overviews or AI Mode. Google-Extended governs one thing only: whether your content is used to train and ground Gemini in Google's other products. The crawler that actually decides whether you appear in Google's AI answers is standard Googlebot. So the well-meaning "block the AI bots" sweep that ends up disallowing Googlebot quietly removes you from Google Search and every Google AI answer at once, while a sweep that blocks only Google-Extended changes nothing about your AI visibility. This post explains what each token really controls, why the mix-up is so common and so costly, how to check your own file in two minutes, and the fix for each case.

What Google-Extended actually controls

Google-Extended is a user-agent token you place in robots.txt, not a crawler that fetches your pages on its own. Google's published guidance is explicit that it governs use, not access: it controls whether content Google has already crawled is used to train Gemini models and to ground Gemini in Google products outside Search. Allowing or blocking it is a training-data decision. It has no bearing on whether your pages can be retrieved and cited inside AI Overviews or AI Mode. Treat it as a training opt-out switch, because that is all it is.

The crawler that actually gates Google's AI answers

Google's own guidance describes AI Overviews and AI Mode as drawing their answers from the ordinary Search index, using a query fan-out across related searches. To be pulled into one of those answers, a page first has to be in that index, and to be in the index it has to be crawlable by standard Googlebot. That makes Googlebot, not Google-Extended, the gate on Google AI visibility. The eligibility bar Google publishes has three parts, and crawlability is the first of them.

“To be eligible to be shown in generative AI features on Google Search, a page must be indexed and eligible to be shown in Google Search with a snippet.”

Read that against what Google-Extended does and the two stop being interchangeable. Googlebot access plus indexing plus snippet eligibility is the path into AI answers. Google-Extended sits entirely outside that path. Disallowing Googlebot fails the first condition for every page at once, which removes you from Google Search and from every Google AI feature built on top of it. Disallowing Google-Extended fails none of them.

Why the mix-up is so common, and so costly

The confusion starts with a reasonable instinct. Operators read that AI companies are scraping the open web, decide they would rather their content did not feed someone else's model, and reach for robots.txt. Google-Extended is a legitimate lever for exactly that, so it goes into the block list. The damage happens when the same sweep also drops a Disallow rule on Googlebot, or a blanket "User-agent: * Disallow: /", in the belief that it is all one "keep the AI bots out" policy. It is not. One line is a harmless training opt-out. The other is the master switch for your entire Google presence, search results and AI answers together.

The two tokens sit a few lines apart in the same file and look almost identical, which is what makes the error so easy to make and so hard to notice. Nothing breaks visibly. Your rank tracker stays green, because classic rankings and a training opt-out are unrelated. The only symptom is a slow, unexplained fade from Google, and by the time anyone connects it to a robots.txt edit made months earlier, a lot of traffic has already gone.

The wider rule that keeps you out of trouble

Google-Extended is one token among several AI-related user agents you will meet. ClarAI's own audit checks five of them by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and OAI-SearchBot. You do not need to memorise what each one does, because every engine publishes its own crawler documentation and that documentation is the authority. You do need one distinction. Some AI crawlers gather content to train models. Others fetch pages live to compose an answer a user is reading right now. Blocking a training crawler protects your content from model training at no visibility cost. Blocking a retrieval crawler removes you from that engine's answers. Before you add any Disallow rule, check the vendor's docs and decide which kind of crawler you are blocking, because the two carry completely different consequences.

How to check your robots.txt in two minutes

• Open yourdomain/robots.txt in a browser. It is plain text, and the whole file is usually short enough to read top to bottom.
• Find the "User-agent: Googlebot" group and scan for any Disallow. A "Disallow: /" under Googlebot, or a global "User-agent: * Disallow: /", is the line that removes you from Google Search and Google AI answers. This is the one that matters most.
• Find the "User-agent: Google-Extended" group. A Disallow here is a training opt-out and nothing more. It is not costing you AI Overviews visibility, so set it however you have genuinely decided on training and move on.
• Look for a blanket "User-agent: * Disallow: /" you did not intend to ship. A single stray global disallow, often left over from a staging config, is the most common self-inflicted outage.
• Confirm in Search Console. URL Inspection shows whether Googlebot can actually fetch and index a given page, which settles any argument about what your robots.txt is really doing.

The fix, decision by decision

1. If you want to stay in Google Search and AI answers: make sure nothing disallows Googlebot. Remove any "Disallow: /" under "User-agent: Googlebot" and any unintended global "User-agent: * Disallow: /".
2. If your only goal was to opt out of AI model training: block Google-Extended, plus the other vendors' training crawlers per their own docs, while leaving Googlebot fully allowed. You keep all of your Search and AI-answer visibility and still keep your content out of training.
3. If you genuinely want out of a specific engine's answers: block that engine's retrieval crawler on purpose, understanding that it removes you from that engine's responses. Make it a decision someone can defend, not a side effect of a copied block list.
4. After any change, wait for a recrawl. robots.txt edits take effect when the crawler next fetches the file, so request reindexing of key URLs through Search Console and expect days rather than minutes.

See it graded, not guessed

ClarAI checks this for you in two places, and both are honest about what a robots.txt check can and cannot tell you. The free AI Readiness Grader at /grade fetches your robots.txt and runs two separate checks on it. Googlebot access is the second-heaviest check on the whole report, worth 20 of 100 points, and AI-crawler access is worth another 15, because these are the binary gates on AI-answer inclusion. If only Google-Extended is blocked, the grader says so plainly: it marks the AI-crawler check as a caution rather than a failure, so it does not collapse your grade the way a Googlebot block does. If Googlebot is disallowed, that is the check that collapses your grade, because that is the real gate. No signup, and no AI cost to run it.

Inside the app, the Tech SEO Audit checks the same five-token AI crawler allowlist plus a dedicated Googlebot check that outranks the rest. A Googlebot disallow is raised as a hard failure. A Google-Extended-only block is deliberately downgraded to a warning rather than a failure, precisely because it does not cost you Google AI visibility, and each finding ships with a copy-ready robots.txt fix. The honest limit worth stating: being crawlable makes you eligible, it does not guarantee you get cited. Passing the crawler checks is the floor, not the finish line, which is why the answer-and-citation monitoring exists alongside them.

Quick answers to the common confusions

• Does blocking Google-Extended hurt my rankings or my AI Overviews? No. Google's published guidance says it only governs Gemini training and grounding in other Google products. Standard Googlebot gates AI Overviews and AI Mode.
• Do I have to allow Google-Extended to appear in AI Overviews? No. Allowing it is not required for Google's AI features. Those depend on Googlebot access plus indexing and snippet eligibility.
• Can I opt out of AI training and still keep AI-answer visibility? Yes. Block the training crawlers, Google-Extended included, and leave Googlebot allowed. The two decisions are independent.
• What is the single most damaging robots.txt line for AI visibility? A "Disallow: /" that applies to Googlebot, whether it sits under "User-agent: Googlebot" or a global "User-agent: *". It removes you from Google Search and every Google AI answer at once.