OpenAI, Google, Microsoft and Anthropic say they will adopt shared anti-scraping signal
🎧 Listen to this story
OpenAI, Google, Microsoft and Anthropic say they will support a common standard that lets websites use their robots.txt files to state whether they allow their content to be used for AI training. The proposal, called a “policy token,” was put forward by Kevin Kelley, Anthropic’s technical lead for standards, through the Internet Engineering Task Force (IETF). The signal would let publishers apply the same preference across multiple companies instead of configuring each one separately.
The Verge reports that the companies’ AI crawlers will stop accessing sites displaying the signal. Anthropic has already adopted it; OpenAI says it will implement the signal in the coming months, while Google and Microsoft plan to support it next year. Microsoft will also offer separate controls for AI training and grounding, but says websites will still appear in Bing search results.
Why it matters · editorial interpretation
A common signal could reduce the work for publishers that would otherwise need to configure content-use preferences separately for each company. Because adoption will happen in stages and Microsoft will retain distinct controls for training and grounding, how those choices are applied may still vary across platforms.