Digital Marketing

One robots.txt Line, Three Different Outcomes

Leo Wang November 9, 2026
One robots.txt Line, Three Different Outcomes

One robots.txt Line, Three Different Outcomes: Google Binds AI Training to Gemini Grounding, Microsoft and OpenAI Keep Them Apart

On 30 October 2026, Google, Microsoft, OpenAI and Anthropic all said they would honor a single robots.txt policy token for AI training preferences. The coverage that followed was close to unanimous: add one line, opt out everywhere, done in ten minutes.

Read Google's own announcement and the ten-minute version falls apart. The sentence that matters is this one: when a site uses the shared token to opt out of training and Google-Extended is not explicitly specified, Google plans to interpret that as if the publisher had disallowed Google-Extended [1].

Now read what Google-Extended controls. Its own documentation says the token governs whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI [2].

One token, two jobs. So on Google, the one-line training opt-out is also a grounding opt-out, and Google's blog states the consequence plainly: websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps [1].

Microsoft published on the same day and made the opposite choice. OpenAI's crawler documentation has worked the opposite way since before any of this. If you add that line without checking which provider does what, you can remove yourself from Gemini's answers while believing you only declined model training.

---

Quick answer

  1. The shared signal is a policy token placed in the User-Agent line of an ordinary robots.txt group. User-Agent: __usage_ai-training followed by Disallow: / expresses a preference against AI training for all paths. No new directives, no new grammar [3].
  2. It covers training only. The IETF vocabulary defines AI Training as modifying the learned parameters of a generative AI model. Grounding-style use sits in a separate category, AI Use, which this token does not bind [4].
  3. Google maps it onto a token that also controls grounding, so a bare training opt-out costs Gemini Apps grounding eligibility unless Google-Extended is explicitly specified [1][2].
  4. Microsoft split training and grounding into two independent preferences, and says neither changes eligibility for Bing search results. You can set training to Disallow and grounding to Allow [5].
  5. OpenAI already separated them by crawler token: GPTBot for training, OAI-SearchBot for search, each setting independent of the others [6].

The boundary on all of it: none of this is in force. Microsoft begins honoring its new preferences on 1 February 2027 and stops reading NoArchive and NoCache as training or grounding preferences after 1 March 2027 [5]. Google aims to support the control in 2027 [1]. The token itself is defined in an individual Internet-Draft that expires on 3 May 2027 [3], and the vocabulary draft it depends on carries a notice that its contents do not reflect working group consensus [4].

---

What the shared token actually is

The mechanism is deliberately unambitious, and that is its best feature. A service operator writes an ordinary robots.txt group whose User-Agent line names a policy token instead of a crawler [3]:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That expresses a preference against AI training for every path except those under /press-kit/, where it expresses a preference in favor. The draft introduces no new directive names and no new grammar: the path language, longest-match precedence and group structure of RFC 9309 are unchanged [3][7]. Every robots.txt parser already deployed will treat the group as one addressed to a user agent it does not recognize, and ignore it.

Four properties of the design decide how you should use it.

It is not a crawling directive. The draft says so in those words. A consumer may use preferences to inform crawling, but if you want to limit crawling you still write groups for crawlers' product tokens. Content-usage preferences and traffic-management signals both live in robots.txt and are independent and orthogonal [3].

Wildcard groups do not count. Consumers MUST NOT interpret User-Agent: * as matching the policy token [3]. Whatever your existing blanket rules say, they express nothing about training.

Silence is not refusal. If no group contains the policy token, no preference has been expressed. The vocabulary models this as a third value, unknown, alongside allowed and disallowed, and takes no position on what default a recipient should assign [3][4].

Contracts outrank it. Private agreements may override preferences expressed through this mechanism, and the vocabulary draft says the same about contractual arrangements [3][4]. A content deal supersedes your file.

---

What "AI training" covers, and what it leaves out

The token corresponds to one category in the IETF vocabulary. Knowing the other two is what keeps you from over-reading the signal.

CategoryLabelWhat it coversBound by the shared token?
AI Trainingtrain-aiUsing an asset to modify the learned parameters of a generative AI model. Training of models used exclusively for non-generative tasks is excluded, even if the model is capable of generative tasksYes
AI Useai-useUsing an asset as input to a generative AI model where the asset is not directly provided by the user. This is where retrieval and grounding sitNo
SearchsearchUsing an asset in an application whose primary purpose is to select assets and direct users to them, where output includes a direct reference or link and excerpts help users judge relevance. Explicitly does not include generating summariesNo

Source: the vocabulary draft, which states that its contents do not reflect working group consensus [4].

Two consequences follow for anyone planning around this.

The exclusion inside the training definition is wider than it looks. A model used or made available for classification, ranking or scoring is not a generative AI model under this vocabulary, even where it generates a rationale for its output [4]. Ranking models are therefore outside the category the token expresses a preference about.

The Search category has an override that will surprise people when it is implemented. Allowing Search permits any processing internal to the application, and the draft spells out that this includes training AI models on the assets, provided the resulting models and their outputs are used exclusively in ways that meet the linking and excerpt conditions. The draft then states that Search overrides any usage falling into other categories, including AI Training and AI Use [4]. The shared token implements only train-ai, so this interaction is not live, but it is the direction the vocabulary is heading.

---

Three providers, three different couplings

This is the table to keep. Every cell comes from the provider's own documentation.

ProviderTraining controlGrounding or search controlAre they coupled?
GoogleGoogle-ExtendedThe same token. Google-Extended governs training of future Gemini models and grounding in Gemini Apps and Grounding with Google Search on Vertex AI [2]Coupled. One switch for both. Separately, Google-Extended does not affect inclusion in Google Search and is not a ranking signal there [2]
MicrosoftAI Training, in robots.txt for a site or path, and also at page level, where the page overrides robots.txtAI Grounding, page level only, with three values: Allow, DisallowExtended, Disallow. There is no site-wide robots.txt grounding preference [5]Independent. Microsoft states that your training setting does not change your grounding setting or the reverse, and that neither changes eligibility for Bing search results [5]
OpenAIGPTBotOAI-SearchBotIndependent. The documentation states each setting is independent of the others, and gives the exact case as its example: allow OAI-SearchBot to appear in search results while disallowing GPTBot for training [6]

Anthropic is absent from that table on purpose. The Internet-Draft was authored by Kevin Kelley of Anthropic [3], and third-party reporting says Anthropic began building support, but I could not read an Anthropic document describing how it maps the token, so this article makes no claim about it. OpenAI's own announcement page returned HTTP 403 when I tried to read it; the OpenAI row above comes from its crawler documentation, which is independent of the announcement and predates it.

Microsoft's grounding middle setting is the one to notice

DisallowExtended has no equivalent anywhere else in the table. It limits grounding use to Microsoft-operated experiences such as AI Mode, AI Overview, Copilot and Microsoft 365, and excludes grounding services Microsoft provides to third parties, such as Web IQ and grounding through API access in Foundry [5].

That is a distinction publishers have been asking for: appear in the assistant your readers use, stay out of the wholesale grounding supply. Microsoft attaches a warning to it in the same post, that selecting DisallowExtended or Disallow will significantly reduce your content's visibility in AI-generated experiences [5].

---

Why Google's mapping is the expensive one

Work through what a brand actually wants. Most companies selling across borders want two things at once: keep their material out of foundation-model training, and keep appearing in AI answers where buyers are looking. On Microsoft and OpenAI those are two separate switches, so you can have both.

On Google they are the same switch, and the published text does not describe a way to split them. Google-Extended governs Gemini training and Gemini grounding together [2], which leaves two positions:

  • Google-Extended disallowed, whether you wrote that yourself or arrived there through the shared token: no Gemini training, and no eligibility to appear and receive clicks via grounding in Gemini Apps [1].
  • Google-Extended allowed: eligible for grounding responses and clicks in Gemini Apps, and content may also be used to train future Gemini models [1][2].

Google's announcement does say that company-specific controls keep working alongside the joint signal, so a site can express a broad choice with the industry token while tailoring instructions for individual providers through controls such as Google-Extended [1]. That is how you stop the shared token from switching Gemini grounding off:

User-Agent: __usage_ai-training
Disallow: /

User-agent: Google-Extended
Allow: /

Be clear about what that buys. It preserves Gemini grounding eligibility, and because Google-Extended covers both uses, it also leaves Gemini training allowed. Google's current text does not describe any combination that expresses "no Gemini training, yes Gemini grounding". That is a real gap in the published description rather than a conclusion about Google's intent, and it is worth asking about before anyone treats the shared token as a complete answer.

One thing the Google mapping does not touch: Google-Extended has no effect on a site's inclusion in Google Search and is not used as a ranking signal there [2]. Classic search visibility is not part of this trade.

---

The dates that are actually on the calendar

Microsoft published a schedule. Google published an aim. Treat them differently.

DateWhat happensSource
15 December 2026Grounding transition preferences open in Bing Webmaster Tools, for sites using NoArchive or NoCache as of that date. Other sites need no transition action[5]
1 February 2027Microsoft begins honoring the new AI Training and AI Grounding preferences, as pages are crawled[5]
1 March 2027Deadline. After this date NoArchive and NoCache are no longer read as a training or grounding preference[5]
3 May 2027The Internet-Draft defining the token expires unless it is progressed[3]
During 2027Google aims to support the shared control alongside other providers[1]
30 September 2027Microsoft's temporary grounding preferences expire, or earlier if the page is recrawled and the new preference is discovered[5]

Two of Microsoft's own caveats belong next to that table. Both its preferences default to Allow when nothing applicable is published, so publishers relying on NoArchive or NoCache should follow the transition steps rather than assume old restrictions carry over. And publishing a setting is not the same as activating it, because Microsoft must crawl the page to discover it [5].

---

Five ways to get this wrong

Treating one line as a complete opt-out. It expresses a preference about training, in the vocabulary's narrow sense, to providers that have implemented it. It blocks nothing, it carries no crawl rule, and on Google it reaches further than its name suggests.

Assuming your existing rules already cover it. A User-Agent: * group MUST NOT be read as matching the policy token [3]. Sites that blocked AI crawlers years ago have expressed no training preference under this mechanism.

Confusing a preference with a control. The vocabulary draft states that preferences are not a security mechanism, and that the specification does not ensure preferences are followed, does not address when they should be followed, and does not address the legal or contractual mechanisms that might make following them mandatory [4].

Citing the drafts as settled standards. The token lives in an individual draft, not a working group draft. The vocabulary it depends on carries an explicit notice that its contents do not reflect working group consensus, and the sections defining the data model and the categories each repeat that [3][4]. Internet-Drafts are explicitly not to be cited as anything other than work in progress.

Acting on the Microsoft timeline while ignoring the Microsoft mechanics. AI Grounding is page level only. There is no site-wide robots.txt grounding preference, so a single robots.txt edit cannot express a grounding choice to Microsoft at all [5].

---

FAQ

If I only do one thing before 2027, what is it?

Find out which of the two positions you are in on Google, because that is the only provider where the choice is forced. Check whether your robots.txt already names Google-Extended. If it does not, and you later add the shared training token without adding Google-Extended, Google plans to read that as a Google-Extended disallow and you lose eligibility to appear and receive clicks via grounding in Gemini Apps [1][2]. Deciding that deliberately is fine. Discovering it afterwards is not.

Does opting out of AI training affect my normal search rankings?

Not according to the two providers that address it directly. Google states that Google-Extended does not impact inclusion in Google Search and is not used as a ranking signal there [2]. Microsoft states that its training and grounding preferences do not change eligibility for Bing search results, subject to normal crawl and indexing requirements [5]. The whole trade sits on the AI-answer side of the house; classic search sits outside it.

Will the shared token stop AI crawlers from hitting my site?

No, and that is by design. The draft states that the policy token is not a directive for crawling behavior. A consumer may use the preference to inform crawling, for instance by not retrieving resources it only wanted for a purpose you declined, but if your goal is to reduce crawl traffic you still write groups for the crawlers' own product tokens [3]. Microsoft adds the converse: an Allow setting does not override a robots.txt rule that blocks Bingbot from crawling [5].

Can I allow training on some sections and refuse it on others?

Yes, and path granularity was an explicit goal of the design. Because the token reuses RFC 9309 path matching and longest-match precedence, Disallow: / with Allow: /press-kit/ expresses a refusal for the site and a permission for that subtree [3][7]. Microsoft also supports the training preference at page level, where the page overrides robots.txt, via a meta tag or an X-Robots-Tag response header [5].

We already have a content deal with one of these companies. Does the token change it?

Probably not, and both drafts say so. The token draft states that private agreements may override preferences expressed through this mechanism, and the vocabulary draft adds that where arrangements such as legal agreements explicitly permit a use, those arrangements likely apply despite a machine-readable statement, unless the arrangement says otherwise [3][4]. Read the contract before editing the file.

---

Bottom line

The engineering here is good. Reusing RFC 9309 wholesale means legacy parsers degrade safely, path rules work the way everyone already understands them, and nobody has to restate existing groups in a file that is size-constrained anyway [3][7]. A single cross-vendor signal for training preferences is a genuine improvement over maintaining a list of proprietary tokens.

The reporting is what went wrong. "One line, opt out everywhere" is true of the syntax and false of the consequences, because the four backers map that line onto very different internal switches. Microsoft and OpenAI keep training separate from grounding and search. Google routes the shared training signal into a token that governs Gemini grounding as well, and says in the same post that grounding eligibility is where the responses and the clicks live.

So the question to settle is not how to add the line. It is this: are you declining to be training data, or declining to be in the answer? On two of these providers you can pick one. On Google, today, the published controls do not let you.

Nothing is in force yet. Microsoft's calendar starts on 15 December 2026 and its deadline is 1 March 2027; Google aims at 2027; the draft that defines the token expires on 3 May 2027. That is enough time to decide on purpose instead of by default.

---

References

[1] - Supporting a shared AI training control for websites — Google Search Central Blog, 30 October 2026

[2] - Google's common crawlers: Google-Extended — Google crawling documentation (scope of the token, covering training of future Gemini models and grounding in Gemini Apps and Grounding with Google Search on Vertex AI)

[3] - Machine-Readable Training Preference Signal — Kevin Kelley (Anthropic), IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00, October 2026, expires 3 May 2027

[4] - A Vocabulary For Expressing AI Usage Preferences — Paul Keller (Open Future) and Martin Thomson (Mozilla, ed.), IETF Internet-Draft draft-ietf-aipref-vocab-08, September 2026 (contents do not reflect working group consensus)

[5] - Your content, your choice: clearer preferences for training and grounding — Krishna Madhavan, Microsoft Bing Blogs, 30 October 2026

[6] - Overview of OpenAI Crawlers — OpenAI developer documentation (GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User, and the independence of each setting)

[7] - Robots Exclusion Protocol — M. Koster, G. Illyes, H. Zeller, L. Sassman, RFC 9309, September 2022

Related Articles