A website has had one way to tell a robot to leave since 1994: a text file named robots.txt. On Oct. 30, four AI companies agreed on one new line for it.
OpenAI, Google, Microsoft and Anthropic said they would honor a shared signal: __usage_ai-training. The line tells every participating company at once to skip a site for AI training1. One company wrote the line. The other three are still finding their keys.
Brian Roemmele@BrianRoemmeleAI Robot - Disallow: / ?
OpenAI, Google, Microsoft, and Anthropic announced on October 30, 2026, that they will honor a single shared signal in robots.txt that lets a site operator state one preference about the use of its content for foundational generative AI training.
The… pic.twitter.com/ScMdrZMdwb
One Engineer Writes the Internet's New Rule
Kevin Kelley holds the title of technical lead for standards at Anthropic. He posted the proposal to an IETF working group himself1.
The filing lists him as its sole author. Its intended status reads "Standards Track." It expires May 3, 20271.
Kelley switched the signal on inside Anthropic the same day he posted it. "I'm also excited to share that Anthropic has already started implementing support for this draft," he wrote to the group. "This complements our existing crawler controls rather than replacing them"1.
Three Companies Sign On, Three Different Clocks
Google, Microsoft and OpenAI each posted their own commitment within hours of Kelley's filing1. All three set a later date instead of Kelley's same-day start.
Microsoft starts honoring it Feb. 1, 20272. That date lands 94 days after Kelley's post. Google's own blog names only "2027," leaving the month open3. OpenAI said it would ship support "in the coming months," leaving its own date just as open1.
Each company still runs its existing crawler rules beside the new one. OpenAI's GPTBot, Google's Google-Extended and Anthropic's ClaudeBot keep working the way they did before Oct. 305. The shared signal adds a single preference a publisher can set once, rather than a replacement for any of them.
Microsoft Waits 94 Days Longer Than Anthropic to Turn the Signal On
Dates each company starts honoring the shared __usage_ai-training robots.txt signal, announced Oct. 30, 2026
Each pin marks the date a company starts honoring the signal; Anthropic's pin sits on announcement day itself.
Google named a year and OpenAI named a season; this chart plots the two companies that each gave one specific date.
Sources: ppc.land [1]; Search Engine Land [2]. Chart by Ryan Elliott Dennis.
The numbers behind this chart
| Date | Event |
|---|---|
| Oct. 30, 2026 | Anthropic: live Oct. 30 |
| Feb. 1, 2027 | Microsoft: live Feb. 1, 2027 |
John Mueller, a Google Search Advocate, wrote the post announcing his company's plan. "Instead of managing separate instructions for every AI developer, website owners will be able to use a shared 'policy token' within robots.txt to set broad preferences across participating AI providers," Mueller wrote3.
Krishna Madhavan, a principal product manager at Microsoft Bing, framed the signal as a fairness fix for publishers caught between two goals. "Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work," Madhavan wrote2.
Only 1 of the 4 Companies Behind the Signal Has Turned It On
Companies that committed to the shared AI training signal on Oct. 30, 2026, by whether they honor it yet
Each cube is one of the four companies behind the signal; the magenta cube is Anthropic, already honoring it, and the three gray cubes are still waiting on their own dates.
Source: ppc.land [1]. Chart by Ryan Elliott Dennis.
The numbers behind this chart
| Item | Value |
|---|---|
| Honoring it now | 1 |
| Still waiting on a date | 3 |
| Whole | 4 |
The Case for Taking the Deal
Before this week, opting out meant finding and configuring a separate rule for every company's crawler. Madhavan's post names that burden directly, and the shared signal removes it with one block of text2.
Mueller's post makes the same trade in Google's own terms. Google already ran its own training opt-out, called Google-Extended, since 20233. The new token layers on top of it. A site that sets the shared signal, and skips a separate Google-Extended rule, automatically receives a Google-Extended block too.
Four companies that compete on almost everything agreed on one piece of text in the same week, a coordination result new to the industry. Meta's communications team called the common signal an important step, short of promising to honor it5.
Search results stay separate from this change. Microsoft says a site keeps appearing in Bing results regardless of its training or grounding preference5. Google's older Google-Extended control has always left search ranking alone too3.
The Case Against Trusting It
Kate Gilchrist runs content and legal operations for the Australian Broadcasting Corporation. She told a parliamentary committee in Sydney on Oct. 6 why she doubts any opt-out system, whether it is shared or solo. "We cannot scour the internet and ensure that we are opting out on all those sites. It simply does not work," Gilchrist testified4.
Gilchrist's committee heard a sharper claim the same day. Research cited there found OpenAI and Anthropic had bypassed robots.txt signals the companies publicly claim to honor4. This week's commitment from both companies leaves that enforcement gap exactly where it was.
The token itself carries only the weight each company gives it by choice. It functions as a request, the kind a company can set aside as easily as it adopted it1. The draft behind it expires in May 2027. Its wording, scope or syntax could still change before any standards body adopts it for real5.
What to do
Add the shared training-opt-out line to your robots.txt file now. Your existing per-company rules stay in force beside it1. Anthropic already reads the new line, so that part of the promise pays off today.
Set a reminder past Feb. 1, 2027, the date Microsoft named, and check back on Google and OpenAI before you drop any older crawler rule. A company still missing a date has yet to keep its half of the promise. Treat the shared signal as one more layer on your existing rules. It supplements them, rather than replacing them, until every company names a day and holds to it.
By the numbers
- 1: Kevin Kelley is the sole author listed on the IETF draft that defines the shared signal1.
- Zero: Anthropic's gap, in days, between Kelley's post and turning the signal on1.
- 94: Microsoft's wait, in days, between Oct. 30 and its Feb. 1, 2027 start date2.
- 2023: Google began offering its own Google-Extended training opt-out this year3.
- May 3, 2027: Kelley's underlying IETF draft expires on this date1.
- 3 of 4: companies behind this week's promise still missing a firm date, OpenAI and Google among them13.
- March 1, 2027: Microsoft's deadline for publishers to swap old NoArchive and NoCache tags for the new preferences5.
What to watch
Watch Google's and OpenAI's blogs for a specific date. A named day would replace "2027" and "coming months," turning a pledge into a schedule.
The IETF working group's own decision matters just as much. It can adopt Kelley's draft as a real standard before May, or let a new one replace it. This claim holds only as long as Anthropic keeps running ahead of the rest. A missed deadline on Anthropic's own side would undercut it just as fast.
Sources
- "Google, Microsoft, OpenAI and Anthropic back one AI training opt-out," ppc.land, Oct. 30, 2026, https://ppc.land/google-microsoft-openai-and-anthropic-back-one-ai-training-opt-out/
- "Microsoft Bing gives publishers more control over how AI uses their content," Search Engine Land, Oct. 30, 2026, https://searchengineland.com/microsoft-bing-gives-publishers-more-control-over-how-ai-uses-their-content-499967
- John Mueller, "Supporting a shared AI training control for websites," Google Search Central Blog, Oct. 30, 2026, https://developers.google.com/search/blog/2026/10/ai-training-control
- "AI Scraping Opt-Out Bypassed by OpenAI and Anthropic, ABC Tells Parliament," Tech Times, Oct. 6, 2026, https://www.techtimes.com/articles/328653/20261006/ai-scraping-opt-out-bypassed-openai-anthropic-abc-tells-parliament.htm
- "Google and Bing back one robots.txt rule to opt out of AI training," ppc.land, Oct. 30, 2026, https://ppc.land/google-and-bing-back-one-robots-txt-rule-to-opt-out-of-ai-training/