OpenAI, Google, Microsoft and Anthropic say they will support a shared robots.txt signal that lets sites opt out of AI training

OpenAI, Google, Microsoft and Anthropic said on October 30 that they will support a shared signal in robots.txt that lets website operators state one AI training preference for several providers at once. The proposal comes from an Internet Engineering Task Force draft, and support is voluntary.

The draft, titled Machine-Readable Training Preference Signal, was written by Kevin Kelley of Anthropic and is dated October 30, 2026. It uses ordinary robots.txt syntax. A site adds a group whose User-Agent line names a policy token, __usage_ai-training, followed by normal Disallow and Allow rules. The draft's own example disallows everything except a press-kit folder. Wildcard groups do not match the token, so a file that does not use it expresses no preference at all. The draft is an individual submission, not an IETF standard, and it expires on May 3, 2027.

Timelines differ. Anthropic says it has already started implementing support. OpenAI says it will honor the signal in the coming months and will share implementation details then. SiliconANGLE reported that Microsoft will add support in the coming months and Google next year. Microsoft also says it is adding a separate AI Grounding preference for pages an AI model fetches to answer a question, and Business Engineer reports Microsoft names February 1, 2027 as the date its new preferences take effect. Meta communications head Andy Stone voiced support for the idea but did not say Meta will honor the token, according to The Verge.

OpenAI said the shared signal will not replace the controls it already offers. GPTBot, OAI-SearchBot and the other agents keep their own rules, and each setting is independent, so a site can allow ChatGPT search while disallowing training. Google says that if a site uses the shared token and does not mention Google-Extended, it plans to treat that as a Google-Extended disallow.

For developers and site owners, the practical gain is less upkeep. Today every AI provider and every crawler needs its own robots.txt entry. One token could replace that list for training, though it records a preference and does not enforce it. The vocabulary draft behind it says it leaves the choice of whether to follow preferences to the recipient. It also does not cover models that perform only non-generative tasks.

If you run a site, nothing changes yet. Keep your existing per-crawler rules, watch for each provider's implementation notes, and decide whether you want to opt out of training, grounding, or both once the token is honored. If you build crawlers or data pipelines, the draft is short and worth reading before you add support.

Building with AI? See this week's hot AI repos and how to use each one.