AI Giants Agree on One ‘Do Not Train’ Signal: What the New Robots.txt Opt-Out Means for Websites

Key takeaway: On October 30, 2026, OpenAI, Google, Microsoft and Anthropic backed a single robots.txt signal that lets websites refuse AI training use with one entry instead of separate rules for every crawler. Anthropic has already started support; Microsoft follows on February 1, 2027, Google in 2027, and OpenAI in the coming months.

What happened

Four of the biggest AI companies announced on October 30, 2026 that they will honor a common signal in robots.txt, the file websites use to tell automated crawlers what they may do. OpenAI said it will honor the signal for “foundational generative AI training,” calling it a voluntary effort that lets publishers communicate the same preference to multiple participating providers at once. Google, Microsoft and Anthropic announced support the same day, according to Engadget and AFP.

The move follows years of friction over bots scraping articles to feed AI models, including The New York Times’ December 2023 lawsuit against OpenAI and Microsoft over alleged unauthorized use of its articles for training. A shared opt-out does not settle those legal fights, but it simplifies the technical side of saying no.

How the new signal works

The proposal centers on one shared “policy token” in robots.txt. A site that wants to refuse AI training use adds a group like this:

User-Agent: __usage_ai-training
Disallow: /

Exceptions are possible too: adding Allow: /press-kit/ permits training use for a press kit while refusing it elsewhere. The technical document is an individual Internet-Draft at the Internet Engineering Task Force (IETF), written by Kevin Kelley, Anthropic’s technical lead for standards, and dated October 30, 2026 — a proposal open for review, not yet an adopted standard.

Old way vs. new way

	Before	With the shared signal
Rules to maintain	One entry per provider, often per crawler (GPTBot, Google-Extended, ClaudeBot)	One entry covering all participating providers
Partial opt-outs	Repeated for each crawler	One group with Allow exceptions
Legal weight	Voluntary crawler controls	Still voluntary; private agreements can override it

When each company will support it

Company	Timeline
Anthropic	Already started
Microsoft	February 1, 2027, plus a separate AI grounding control
Google	In 2027; applies to Google-Extended, its AI training crawler
OpenAI	In the coming months; existing controls stay in place meanwhile

Support is staggered, not simultaneous, so a publisher adding the token today should not assume all four companies honor it yet. Meta has expressed support for the idea without committing to implement it, according to reporting on the announcements.

What it does not do

• It does not stop crawling. The token expresses a training preference only; it does not ban bots from visiting a site or remove pages from search results.
• It does not undo past scraping. Content already collected stays collected; the signal governs future training use only.
• It is opt-out, not opt-in. Training scraping remains allowed by default until a publisher changes the setting, Microsoft said.
• It is not a payment deal. The announcements set out implementation plans, not compensation for publishers.

Training vs. grounding: Microsoft’s extra split

Microsoft plans a separate control for AI grounding: when a model fetches a page to inform an answer, rather than to gather training data. A site could therefore refuse training use while still letting its pages inform AI answers, or refuse both. That distinction matters because grounding drives visibility in AI answers today, while training shapes future models — so it deserves a deliberate choice, not a reflex block. See our Google AI Mode and SEO strategy guide for why that visibility trade-off matters.

What does it cost?

Nothing in money terms: robots.txt is a plain text file every site already serves. The real cost is attention — deciding the policy, applying it, keeping existing per-crawler rules until each company confirms support, and re-checking later that the token is honored.

What you need before you use it

Access to edit your robots.txt file, a decision on scope (whole site or exceptions such as a press kit), and patience: implementation is staggered and OpenAI has promised further details. Keep existing GPTBot and Google-Extended blocks in place until every provider you care about confirms support.

Limits and risks

The main risk is treating a voluntary, draft-stage signal as a guarantee. Compliance is not yet independently verifiable, not every AI provider has signed on, and the draft says private agreements may override the preference. Refusing grounding, where offered, could also reduce how often a site’s content informs AI answers. Our trust layer for AI explainer covers the broader controls behind this.

Knowns, unknowns, and what to watch

Known: four major AI companies have committed to the __usage_ai-training token, and Anthropic has already started support. Unknown: each company’s detailed implementation rules, how compliance will be verified, and whether other providers will commit. What to watch: Microsoft’s February 1, 2027 deadline, Google’s 2027 rollout, OpenAI’s promised update, and whether the IETF draft advances toward a standard.

Frequently asked questions

What is the new AI training opt-out signal?

It is a shared robots.txt entry using the policy token __usage_ai-training. A site adds that user agent with Disallow: / to refuse generative AI training use, and OpenAI, Google, Microsoft and Anthropic have committed to honor it.

Does adding the signal remove my site from Google Search or ChatGPT answers?

No. The signal covers AI training use only. It does not block crawling in general or remove pages from search results or AI answers. Microsoft’s separate grounding control covers the AI-answers case; the shared token does not.

When will the signal actually work?

Anthropic has already started support. Microsoft begins on February 1, 2027, Google plans support in 2027, and OpenAI aims to implement it in the coming months. Until each company confirms its rollout, treat the token as a stated preference, not an enforced block.

Does it delete content companies already collected?

No. The signal applies to future training use only and cannot undo past scraping. Content crawled before a publisher adds the token is unaffected.

Do I still need my existing robots.txt rules?

Yes, for now. OpenAI says its existing crawler controls will continue to apply, and support for the shared token is staggered. Keep rules such as GPTBot and Google-Extended blocks until every provider you care about confirms it honors the new token.

Sources: OpenAI, “Making it easier for websites to express AI training preferences,” October 30, 2026; Engadget, “Google Joins OpenAI And Anthropic In Promising New Scraping Controls,” October 31, 2026; AFP via Channel News Asia, “AI giants promise publishers new way to block some scraping,” October 31, 2026; IETF Internet-Draft draft-kelley-aipref-training-preference-signal, October 30, 2026.