Anthropic drafts a shared robots.txt signal for opting out of AI training, and Google and OpenAI say they will honour it
A single token, __usage_ai-training, would let a website say no to AI training once, in its ordinary robots.txt file, instead of blocking each company's crawler.
On 30 October Kevin Kelley of Anthropic published an Internet-Draft at the IETF, the body that sets internet standards, called “Machine-Readable Training Preference Signal”. It lets a website state whether its pages may be used to train AI models through its existing robots.txt file, using a token named __usage_ai-training, without adding any new syntax. The draft says: “This policy token is not a directive for crawling behavior.” In other words, it tells companies what they may do with the pages, not whether they may fetch them.
The same day Google added a note to its crawler documentation changelog about its intention to support the shared training-preference signal. OpenAI published a post on making it easier for websites to express AI training preferences; its news feed summary says OpenAI will honor a shared robots.txt signal. PPC Land reported that Microsoft also backs it, and that Google will treat an opt-out as if the site had blocked Google-Extended, its existing AI-training token.
Until now a publisher who wanted to keep its pages out of AI training had to find and block each company's crawler by name, and the names keep changing. A shared token could make one line in robots.txt do the job for every company that agrees to honour it.
The draft is not yet a standard, and it binds only the companies that choose to follow it. Google's changelog entry gives no date; PPC Land reported that Google plans support in early 2027, per its crawler documentation, and OpenAI in the coming months.
- Confirmed Anthropic's Kevin Kelley published the Internet-Draft “Machine-Readable Training Preference Signal”, which uses the robots.txt token __usage_ai-training. IETF, draft-kelley-aipref-training-preference-signal-00
- Confirmed Google's crawler changelog for 30 October notes Google's intention to support the shared machine-readable training preference signal. Google Search Central crawling changelog
- Claimed OpenAI says it will honor a shared robots.txt signal for AI training preferences. OpenAI news feed
- Reported Microsoft also backs the signal, and Google will treat an opt-out as a Google-Extended block, PPC Land reported. PPC Land
Courts & policySafety, security & governance
Catching up in the November 2, 2026 edition · front page