On Oct. 30, 2026, Google, OpenAI and Microsoft each said they will honor a shared preference signal in robots.txt so publishers can express AI training choices once instead of maintaining separate rules for every major provider. The work tracks the IETF AIPREF working group and Internet-Draft Machine-Readable Training Preference Signal.
Key facts
| Item | Detail |
|---|---|
| Signal | __usage_ai-training policy token in robots.txt |
| Backed by | Google, OpenAI, Microsoft |
| Basis | IETF AIPREF Internet-Draft |
| Timing | OpenAI coming months; Google aims 2027; Microsoft from Feb. 1, 2027 |
What was announced
Google Search Central said it plans to support a shared AI training control in robots.txt built on AIPREF work, extending controls such as Google-Extended that Google has offered since 2023. OpenAI said it will honor a common robots.txt signal for foundational generative AI training. Microsoft's Bing team is adopting the shared training preference and adding page-level controls for training and AI grounding in Bing-powered experiences. Each company published its own timeline for honoring the signal.
How the robots.txt signal works
The draft and Microsoft's examples use existing robots.txt grammar with a shared user-agent token rather than new directive names. A site-wide opt-out of AI training can look like this:
User-Agent: __usage_ai-training
Disallow: /Path-level rules work like ordinary allow and disallow lines. The Internet-Draft's sample also shows an exception such as Allow: /press-kit/ under that user agent. The document is an Internet-Draft—not a finished standard—and expires 3 May 2027.
Google's plan
Google said owners can use a shared policy token to set broad preferences across participating providers. If a site opts out via that token without explicitly specifying Google-Extended, Google plans to treat the choice as if Google-Extended were disallowed. Company-specific controls keep working alongside the joint signal.
Google noted alignment with the EU Code of Practice for GPAI under the EU AI Act, which recognizes robots.txt for machine-readable preferences. Google aims to support the new control with other AI providers in 2027.
OpenAI's plan
OpenAI said participation is voluntary and that it will honor the common signal for foundational generative AI training, aiming to implement it in the coming months. Existing crawler controls continue to apply.
Microsoft: training, grounding and 2027 dates
Microsoft extends training preferences to the page level with <meta name="Bingbot" content="ai-training:disallow"> or X-Robots-Tag: Bingbot: ai-training:disallow, plus a separate AI grounding preference (Allow, DisallowExtended, Disallow). DisallowExtended limits grounding to Microsoft-operated experiences such as AI Mode, AI Overview, Copilot and Microsoft 365, excluding third-party grounding like Web IQ and Foundry's grounding API. Both preferences default to Allow; Bing search eligibility is unchanged.
NoArchive and NoCache stop serving as training or grounding signals after March 1, 2027. Microsoft honors the new preferences from Feb. 1, 2027. Disallow settings can reduce AI visibility; Microsoft says neither choice guarantees inclusion, traffic or payment.
What it does not do
The shared signal covers training use—and, for Microsoft, grounding—not ordinary crawling or traditional search indexing. Opting out of AI training does not remove pages from blue-link results. Publishers still need provider-specific rules where the shared token is not enough.
What it means for site owners
Analysis (ours). If you already maintain Google-Extended, GPTBot or Bing-specific lines, treat Oct. 30 as coordination—not an emergency rewrite. Decide whether a single __usage_ai-training block plus exceptions matches your policy, and keep per-provider overrides where needed. Court fights over training data continue—see Canadian Newspapers Sue Anthropic Over Claude Training Data—while these vendors align on one robots.txt vocabulary. Technical signals and litigation are parallel tracks.
What to watch
Adoption by other AI developers, OpenAI's rollout timeline, Google's 2027 support, and Microsoft's Feb. 1 and March 1, 2027 cutovers. Watch IETF AIPREF progress before the draft expires in May 2027.
Sources
- Google Search Central Blog: Supporting a shared AI training control for websites — Oct. 30, 2026
- OpenAI: Making it easier for websites to express AI training preferences — Oct. 30, 2026
- Microsoft Bing Blogs: Your content, your choice: clearer preferences for training and grounding — Oct. 30, 2026
- IETF Datatracker: Machine-Readable Training Preference Signal — Oct. 30, 2026