OpenAI, Google, Microsoft and Anthropic say they will support a proposed AI scraping opt out signal that would let website owners state AI-training preferences once rather than repeat the instruction for each company’s crawler. The promise is useful only to the extent each company actually implements and honors it, but it could remove some of the current crawler-by-crawler administrative nonsense.
The proposal comes from Kevin Kelley, Anthropic’s technical lead for standards, through the Internet Engineering Task Force. It describes a shared “policy token” placed in a site’s robots.txt file, according to The Verge. That file has long been used to communicate crawler-access preferences, including for search indexing, archiving and AI-related collection.
How would the AI scraping opt-out signal work?
Publishers currently need to express their AI-training choices separately to individual providers. The proposed token is meant to give multiple AI companies one common instruction. The Verge reported that OpenAI, Google, Microsoft and Anthropic said their AI crawlers would restrict access when they encounter the shared token.
That is a proposed technical convention, not a legal ban on scraping or a guarantee that every crawler will comply. The IETF proposal also is not the same thing as a completed, universally adopted standard.
Which companies have committed to use it?
- Anthropic: already adopted the signal, The Verge reported.
- Google: says it plans to support it next year.
- Microsoft: also plans support next year.
- OpenAI: says it will implement the signal in the coming months.
- Meta: communications chief Andy Stone welcomed the idea of a common signal and said the company wanted to work with others on standards and implementation. He did not say Meta would honor the setting.
Microsoft said its planned controls will distinguish between opting out of AI training and opting out of AI grounding. Training uses material in developing models. Grounding points a model to verified information sources to help it answer a request. Microsoft said either preference would not remove a site from Bing search results. The Verge also reported that Google already offers related controls for AI search features and AI training that do not affect ordinary search visibility.
Publishers have been trying other responses to AI crawlers. The Verge pointed to the Really Simple Licensing standard, which is intended to let publishers charge crawlers, and Cloudflare’s default blocking of AI crawlers. Those approaches address access or payment differently from a shared preference signal.
Meta remains separate from this rollout. Nieman Lab reported that Meta’s commercial terms effective October 30 authorize the company to crawl and collect content from sites associated with business accounts, potentially for training among other listed uses. A Meta spokesperson told Nieman Lab that owners could manage access with robots.txt. That statement does not establish that Meta will honor the proposed shared token.
This story draws on original reporting from The Verge.
