Google-Extended Does Not Do What You Think It Does

Blocking Google-Extended in your robots.txt does not remove your brand from Google's AI Overviews or AI Mode. Google-Extended governs how Google may use content it has already crawled: training Gemini and grounding answers in the Gemini app and Vertex AI, and, per Search Console's help, the training of the models behind Search's generative AI features. It does not decide whether you appear. The crawler that actually decides whether you appear in Google's AI answers is standard Googlebot. So the well-meaning "block the AI bots" sweep that ends up disallowing Googlebot quietly removes you from Google Search and every Google AI answer at once, while a sweep that blocks only Google-Extended leaves Google Search, AI Overviews and AI Mode untouched, though it does stop Google using your pages to ground answers in the Gemini app. This post explains what each token really controls, why the mix-up is so common and so costly, how to check your own file in two minutes, and the fix for each case.

What Google-Extended actually controls

Google-Extended is a user-agent token you place in robots.txt, not a crawler that fetches your pages on its own. Google's published guidance is explicit that it governs use, not access. Google's common crawlers page says it controls whether content Google has already crawled is used to train future Gemini models, which power the Gemini app and the Vertex AI API for Gemini, and to ground answers in the Gemini app and in Grounding with Google Search on Vertex AI. Search Console's help adds that it is also the control for limiting training of the models used to generate responses in Search's generative AI features. Google states it does not affect whether a site is included in Google Search and is not a ranking signal, so it has no bearing on whether your pages can be retrieved and cited inside AI Overviews or AI Mode. Treat it as a switch for training and for Gemini app grounding, not as a visibility switch for Google Search. (Corrected 13 October 2026: this paragraph called Google-Extended a training opt-out and nothing more, and said it only covers Google products outside Search. Google's pages say it also controls grounding in the Gemini app and Vertex AI, and Search Console's help says it limits training of the models behind Search's AI features. Our quick answers below were corrected the same way.)

The crawler that actually gates Google's AI answers

Google's own guidance describes AI Overviews and AI Mode as drawing their answers from the ordinary Search index, using a query fan-out across related searches. To be pulled into one of those answers, a page first has to be in that index, and to be in the index it has to be crawlable by standard Googlebot. That makes Googlebot, not Google-Extended, the gate on Google AI visibility. The eligibility bar Google publishes has three parts, and crawlability is the first of them. (Updated 1 October 2026: since 31 August 2026 Search Console also has a site-wide Search generative AI setting; include is the default, and exclude removes a site from AI Overviews and AI Mode whatever its robots.txt says.)

“To be eligible to be shown in generative AI features on Google Search, a page must be indexed and eligible to be shown in Google Search with a snippet.”

Read that against what Google-Extended does and the two stop being interchangeable. Googlebot access plus indexing plus snippet eligibility is the path into AI answers. Google-Extended sits entirely outside that path. Disallowing Googlebot fails the first condition for every page at once, which removes you from Google Search and from every Google AI feature built on top of it. Disallowing Google-Extended fails none of them.

Why the mix-up is so common, and so costly

The confusion starts with a reasonable instinct. Operators read that AI companies are scraping the open web, decide they would rather their content did not feed someone else's model, and reach for robots.txt. Google-Extended is a legitimate lever for exactly that, so it goes into the block list. The damage happens when the same sweep also drops a Disallow rule on Googlebot, or a blanket "User-agent: * Disallow: /", in the belief that it is all one "keep the AI bots out" policy. It is not. One line is a harmless training opt-out. The other is the master switch for your entire Google presence, search results and AI answers together.

The two tokens sit a few lines apart in the same file and look almost identical, which is what makes the error so easy to make and so hard to notice. Nothing breaks visibly. Your rank tracker stays green, because classic rankings and a training opt-out are unrelated. The only symptom is a slow, unexplained fade from Google, and by the time anyone connects it to a robots.txt edit made months earlier, a lot of traffic has already gone.

The wider rule that keeps you out of trouble

Google-Extended is one token among several AI-related user agents you will meet. ClarAI's own audit checks seven of them by name and labels each by role: the retrieval agents OAI-SearchBot, PerplexityBot, Claude-SearchBot and Claude-User, and the training crawlers GPTBot and ClaudeBot plus the Google-Extended policy token. You do not need to memorise what each one does, because every engine publishes its own crawler documentation and that documentation is the authority. You do need one distinction. Some AI crawlers gather content to train models. Others fetch pages live to compose an answer a user is reading right now. Blocking a training crawler protects your content from model training at no visibility cost. Blocking a retrieval crawler removes you from that engine's answers. Before you add any Disallow rule, check the vendor's docs and decide which kind of crawler you are blocking, because the two carry completely different consequences.

How to check your robots.txt in two minutes

• Open yourdomain/robots.txt in a browser. It is plain text, and the whole file is usually short enough to read top to bottom.
• Find the "User-agent: Googlebot" group and scan for any Disallow. A "Disallow: /" under Googlebot, or a global "User-agent: * Disallow: /", is the line that removes you from Google Search and Google AI answers. This is the one that matters most.
• Find the "User-agent: Google-Extended" group. A Disallow here opts you out of Gemini training and of grounding in the Gemini app. It is not costing you AI Overviews or AI Mode visibility, so set it however you have genuinely decided on training and the Gemini app, and move on.
• Look for a blanket "User-agent: * Disallow: /" you did not intend to ship. A single stray global disallow, often left over from a staging config, is the most common self-inflicted outage.
• Confirm in Search Console. URL Inspection shows whether Googlebot can actually fetch and index a given page, which settles any argument about what your robots.txt is really doing.

The fix, decision by decision

1. If you want to stay in Google Search and AI answers: make sure nothing disallows Googlebot. Remove any "Disallow: /" under "User-agent: Googlebot" and any unintended global "User-agent: * Disallow: /".
2. If your only goal was to opt out of AI model training: block Google-Extended, plus the other vendors' training crawlers per their own docs, while leaving Googlebot fully allowed. You keep your Google Search, AI Overviews and AI Mode visibility and still keep your content out of training; the one trade-off is that Google stops using your pages to ground answers in the Gemini app.
3. If you genuinely want out of a specific engine's answers: block that engine's retrieval crawler on purpose, understanding that it removes you from that engine's responses. Make it a decision someone can defend, not a side effect of a copied block list.
4. After any change, wait for a recrawl. robots.txt edits take effect when the crawler next fetches the file, so request reindexing of key URLs through Search Console and expect days rather than minutes.

See it graded, not guessed

ClarAI checks this for you in two places, and both are honest about what a robots.txt check can and cannot tell you. The free AI Readiness Grader at /grade fetches your robots.txt and runs two separate checks on it. Googlebot access is the second-heaviest check on the whole report, about 22 percent of a fully measured grade (20 of the 91 points across the seven scored checks), and access for the retrieval crawlers that build AI answers is about 16 percent (15 of 91), because these are the binary gates on AI-answer inclusion. If only Google-Extended is blocked, the grader says so plainly: it shows the block as a caution on a separate training-crawler row that is informational and carries no points, so it does not change your grade at all, unlike a Googlebot block. If Googlebot is disallowed, that is the check that collapses your grade, because that is the real gate. No signup, and no AI cost to run it. Updated 19 October 2026: the grader's Markdown-for-agents row stopped scoring and is now shown for information only, so these shares are now out of 91 points across seven scored checks.

Inside the app, the Tech SEO Audit checks the same seven role-labelled tokens plus a dedicated Googlebot check that outranks the rest. Since September 2026 it also reads Bingbot, Applebot and the user-triggered ChatGPT-User and Perplexity-User fetchers, applies robots.txt rules per page path, and tests each page with crawler user agents to tell you if your firewall or CDN refuses them (a test of user-agent rules, not a guarantee about the real crawlers). A Googlebot disallow is raised as a hard failure. A Google-Extended-only block is deliberately downgraded to a warning rather than a failure, precisely because it does not cost you AI Overviews or AI Mode visibility, and each finding ships with a copy-ready robots.txt fix. The honest limit worth stating: being crawlable makes you eligible, it does not guarantee you get cited. Passing the crawler checks is the floor, not the finish line, which is why the answer-and-citation monitoring exists alongside them.

Quick answers to the common confusions

• Does blocking Google-Extended hurt my rankings or my AI Overviews? No. Google's published guidance says it governs Gemini training, grounding in the Gemini app and Vertex AI, and the training of the models behind Search's AI features, and that it does not affect inclusion in Google Search or ranking. Standard Googlebot gates AI Overviews and AI Mode.
• Do I have to allow Google-Extended to appear in AI Overviews? No. Allowing it is not required for Google's AI features. Those depend on Googlebot access plus indexing and snippet eligibility.
• Can I opt out of AI training and still keep AI-answer visibility? Mostly. Block the training crawlers, Google-Extended included, and leave Googlebot allowed: you keep Google Search, AI Overviews and AI Mode. The trade-off is the Gemini app, because Google-Extended also controls grounding there, so blocking it stops Google using your pages to ground Gemini app answers.
• What is the single most damaging robots.txt line for AI visibility? A "Disallow: /" that applies to Googlebot, whether it sits under "User-agent: Googlebot" or a global "User-agent: *". It removes you from Google Search and every Google AI answer at once.