Google, OpenAI and Microsoft Back a Shared AI Training Opt-Out

Website owners will be able to refuse AI training by several of the largest AI developers with one AI training opt-out in robots.txt, after Google, OpenAI and Microsoft each said on 30 October that they plan to honour a shared signal. It is a voluntary industry step, not a law, and none of the three has switched it on yet. OpenAI says it aims to implement the signal “in the coming months”, Google says it aims to support it “in 2027”, and Microsoft says its crawler will start honouring the new preferences from 1 February 2027.

Until now, a site that wanted to keep its pages out of AI training had to find and name each company’s crawler or control, such as Google-Extended for Google. The new approach puts a single “policy token” in robots.txt that any participating AI developer can read. It arrives while the terms on which AI firms use web content are being tested in courts and competition cases, including the UK’s CMA investigation into Meta’s terms.

How the AI training opt-out works in robots.txt

The syntax comes from an Internet-Draft published on 30 October, draft-kelley-aipref-training-preference-signal-00. It defines a policy token, __usage_ai-training, written with two leading underscores, which sits in the User-Agent line of an ordinary robots.txt group. A site that wants to opt out of training for all its pages would add:

User-Agent: __usage_ai-training
Disallow: /

Path rules work the same way. The draft’s own example opts a whole site out but allows a press-kit folder. It says the mechanism adds “no new directive names and no new grammar” to robots.txt, so older parsers that lack support for the token simply skip it.

Two details matter for anyone editing the file. A catch-all User-Agent: * group is not read as matching the token, so an existing blanket rule says nothing about training either way. And the token governs use, not access: “This policy token is not a directive for crawling behavior,” the draft says. Blocking a crawler is still done with that crawler’s own name.

The draft borrows its definition from the IETF’s AI Preferences (AIPREF) working group, where training means “using an asset to modify the learned parameters of a generative AI model”. Models used only for non-generative tasks, such as ranking or classification, fall outside it.

The document has no formal standing yet. The IETF datatracker lists it as an individual draft that “is not endorsed by the IETF and has no formal standing in the IETF standards process”. Its author, Kevin Kelley, is listed with Anthropic as his affiliation, and the draft expires on 3 May 2027 unless it is replaced.

What each company has committed to

OpenAI said it “will honor a common signal in robots.txt” for “foundational generative AI training”, calling it a “voluntary effort”. The signal “will not replace the controls we already provide through robots.txt”, it said, so its existing crawler rules stay in force. It added: “We encourage other AI providers to honor the signal.”

Google said that when a site uses the shared token to opt out of training “and Google-Extended is not explicitly specified”, it plans to treat that as if Google-Extended had been disallowed. Company-specific controls keep working alongside it. Google called the proposal “consistent with” the EU’s Code of Practice for general-purpose AI models under the AI Act, which recognises robots.txt as a main way to state machine-readable preferences. The post was written by John Mueller, a Google Search Advocate.

Microsoft went further. In a post by Krishna Madhavan, principal product manager for Microsoft Bing, it said it will read the shared training token and also offer a separate page-level “AI Grounding” preference, covering whether content can help shape AI answers in products such as Copilot and Microsoft 365. A “DisallowExtended” setting limits grounding to Microsoft’s own products and excludes the grounding services it sells to other companies. Sites that rely on its older NoArchive and NoCache tags have until 1 March 2027 to switch; after that, those tags “will no longer be read as a preference for training or grounding”. Microsoft says none of these choices changes eligibility for ordinary Bing results, and that “Neither choice guarantees inclusion, traffic, or payment.”

Why a shared AI training opt-out still leaves gaps

The appeal is plain. Site owners have spent two years chasing a lengthening list of crawler names, and one line is easier to maintain than a dozen. Because the token reuses robots.txt grammar, it costs almost nothing to adopt for anyone who already keeps that file. Crawling has become a negotiation in other ways too, as the Cloudflare and OpenAI network-signal pilot showed.

The limits are just as clear. The signal is a request, not a legal right. Nothing in the three announcements makes it binding, and the draft says private agreements can override it. It also looks forward only: it applies as companies crawl after switching it on, and none of the announcements mentions removing material already used. Only participating firms will read it, and OpenAI’s call for “other AI providers” to join amounts to an admission that coverage is incomplete. Meta’s dispute over its crawling terms shows how a large company can still write its own rules.

The bigger issue for news sites is the split between training and answering. Google ties Gemini grounding to allowing Google-Extended, and Microsoft warns that blocking grounding will reduce visibility in AI answers. That mirrors the line in Wikimedia’s court brief, which argued that training is likely fair use but retrieval-based copying is not. An AI training opt-out settles only half of what site owners care about: whether AI tools may summarise their work for readers who never click through. That question sits at the centre of the CMA’s Meta case, and no robots.txt line answers it.

What happens next

The working group’s wider vocabulary is still under discussion, and Google says it will “continue to work with these stakeholders to refine and standardize this through the IETF working group”. Microsoft has given the first firm dates: temporary grounding settings open in Bing Webmaster Tools on 15 December 2026 for sites using NoArchive or NoCache, and its crawler starts honouring the new preferences on 1 February 2027. OpenAI has promised another update, with implementation details, when it ships.

Sources

Primary:

• OpenAI, “Making it easier for websites to express AI training preferences”, 30 October 2026
• Google Search Central Blog, “Supporting a shared AI training control for websites”, 30 October 2026
• Microsoft Bing Blog, “Your content, your choice: clearer preferences for training and grounding”, 30 October 2026
• IETF datatracker, draft-kelley-aipref-training-preference-signal-00, “Machine-Readable Training Preference Signal”, 30 October 2026

Press:

• None.

Related stories

• Meta Delays Its Crawling Terms to 20 November as Publishers Object
• Wikimedia Brief Says AI Training Is Likely Fair Use, but RAG Is Not
• UK Watchdog Opens Competition Case Over Meta’s Terms for Publishers