AI Giants Unify Opt-Out Protocols for Web Scraping

The proposal, spearheaded by Anthropic’s technical lead Kevin Kelley, is currently moving through the Internet Engineering Task Force. This move addresses a growing friction point for digital publishers who have struggled to manage the deluge of automated traffic harvesting content for large language models. Previously, sites were forced to navigate a complex patchwork of proprietary opt-out requirements for every major AI developer.

While Anthropic has already integrated the signal, implementation timelines vary across the industry. OpenAI expects to adopt the standard within the coming months, while Google and Microsoft have slated their support for next year. Microsoft intends to layer additional nuance onto the protocol, allowing publishers to distinguish between general AI training and AI grounding—the mechanism used to anchor model responses in verified data. Crucially, the company clarified that opting out of training will not impact a site's visibility within Bing’s search index.