OpenAI, Google, Microsoft, and Anthropic unite behind a universal AI scraping opt-out standard
The four leading AI labs endorse a shared robots.txt token that lets publishers block AI training across all companies at once, sidestepping the need for individual takedown requests.
Last verified:
A unified signal for AI content permissions
Four of the world’s largest AI companies—OpenAI, Google, Microsoft, and Anthropic—are converging around a single technical standard that lets publishers block their content from being scraped for AI training without negotiating separate agreements with each firm. According to The Verge AI, the approach uses a shared “policy token” embedded in the robots.txt file, the decades-old text document that websites use to communicate crawler permissions to search engines and other bots. Under this new regime, a single line added to robots.txt can signal preferences to multiple AI trainers at once, replacing the current patchwork where publishers must file individual takedown requests or negotiate directly with each company.
How the standard works and where it lives
The policy token proposal originates from Kevin Kelley, a technical lead for standards at Anthropic, working through the Internet Engineering Task Force (IETF). By placing the signal within robots.txt—infrastructure that has governed web scraping for over two decades—the initiative sidesteps the need for new infrastructure or compliance frameworks. According to The Verge AI, OpenAI, Google, Microsoft, and Anthropic have each committed to respecting this signal by restricting their AI crawlers when they encounter it. Microsoft is adding a refinement: granular controls that distinguish between blocking AI training and blocking “AI grounding,” the process where AI models are anchored to cited sources to reduce hallucinations. This distinction allows publishers to permit one form of AI use while blocking another.
Implementation timeline and current rollout status
Anthropic has already begun honoring the policy token, according to The Verge AI. Google and Microsoft expect to roll out support within 2027, while OpenAI says it will implement the standard “in the coming months” from the October 30th publication date. Meta communications head Andy Stone signaled openness to the standard—remarking that “creating a common signal that conveys such preferences to multiple providers at once is an important step”—but stopped short of announcing a firm implementation date. The Verge AI updated its report to include Stone’s statement on October 30th, clarifying Meta’s position as supportive but non-committal.
Publisher control and search visibility safeguards
A critical design detail: both Google and Microsoft have publicly confirmed that opting out of AI training or grounding does not penalize a site’s appearance in traditional search results. According to The Verge AI, Google previously implemented similar controls allowing sites to exclude their content from AI search features without losing organic search visibility. Microsoft extends this promise to Bing, stating that websites will continue to appear in search results regardless of their AI training preferences. This distinction addresses a key concern among publishers—that refusing to feed AI companies would harm their search engine rankings.
Context: The escalating scraping and licensing landscape
The policy token arrives amid escalating tensions between AI companies and content creators. According to The Verge AI, publishers have deployed a range of countermeasures: Really Simple Licensing (RSL) lets publishers charge AI crawlers per visit, while Cloudflare has begun blocking AI bots by default. The new standard does not preclude these other approaches; instead, it establishes a baseline, machine-readable opt-out that works across the largest AI training operations simultaneously.
Why this matters
For publishers tired of fielding individual scraping notices and takedown requests, this standard dramatically reduces coordination costs—one robots.txt edit broadcasts a unified preference to four major AI labs. For AI companies, the signal reduces legal ambiguity and goodwill erosion by providing a clear, technical path for publishers to assert their preferences. However, the standard’s real-world effectiveness depends on adoption. Meta’s hesitation and the delayed rollout timelines (2027 for Google and Microsoft) suggest that enforcement will remain uneven for at least the next year. Small publishers may still need to monitor multiple company-specific opt-out portals, while only the largest firms can rely on the policy token as a catch-all. The initiative is a step toward legitimacy but not yet a replacement for negotiation.
Frequently Asked Questions
How does this policy token differ from existing robots.txt rules?
Traditional robots.txt controls search engine indexing. The new policy token extends this to AI-specific permissions, allowing publishers to block AI training, AI grounding, or both—without affecting search visibility or archiving.
Will Meta honor this standard?
Meta communications head Andy Stone signaled support for the initiative but did not commit to implementation. The statement was added as an update on October 30th.
When will each company implement this?
Anthropic has already adopted the signal. Google and Microsoft plan to support it in 2027, while OpenAI says it will implement the standard in the coming months.
Can publishers still appear in AI search features if they opt out?
Google and Microsoft both confirm that opting out of AI training does not affect a site's visibility in traditional search results or search-based AI features like Bing or Google's AI overviews, depending on the granularity of the opt-out.