Last week, Google, Microsoft, Anthropic, and OpenAI each pledged support for a fresh technical approach that would enable website publishers to signal their preference against having their material incorporated into AI model training datasets. The initiative, developed by Anthropic's Kevin Kelley, generated substantial media interest and social media discussion, with many outlets emphasizing the potential for publishers to gain more say in how their work is used by artificial intelligence companies.
Yet the announcements—and much of the ensuing commentary—omit crucial details about how the product changes connect to ongoing standardization efforts and the wider policy landscape. Grasping these distinctions demands examining the standards development process underway and the larger policy conversations around AI, search, and market competition.
The current proposal emerges from years of friction between content publishers and web-crawling firms, playing out in litigation, standards bodies, and public debate. Roughly two years back, the Internet Engineering Task Force (IETF), the leading body for internet standards creation, initiated an initiative called AI Preferences (AIPREF) to build new standards offering websites broader tools for automatically communicating their wishes about how their material gets deployed for AI purposes. The effort builds partly on an earlier IETF standard, RFC 9309: Robots Exclusion Protocol, which defines the format for the widely used robots.txt files that websites employ to manage crawler behavior. Anthropic, Google, and Microsoft have all taken active roles in the AIPREF group, whereas OpenAI's involvement has been limited, despite the company's claim last week that the announcement "strengthens our continued participation in the IETF standards discussions."
The AIPREF initiative was prompted in part by work on the EU General-Purpose AI Code of Practice, a voluntary compliance framework under the EU AI Act. Its signatories—predominantly the largest American and European AI model developers—pledge to respect the machine-readable preferences that websites communicate.
1. The mechanism carries no official standard status—yet backing from the four dominant search and AI firms could make it a de facto standard.
The proposal unveiled last week holds no formal standing within the IETF. It will require discussion and deliberation—potentially extensive—to determine whether it advances toward formal standardization. However, when the largest companies roll out support for it (all have committed to do so by 2027), formal standardization status may prove irrelevant. Should website operators adopt the new proposal and modify their robots.txt files accordingly, it could become the practical standard for signaling AI training preferences regardless of the standards process outcome.
The proposal released on October 30 (referred to informally as the "policy token draft") exists as what the IETF community calls an "individual draft"—a submission by a single author with no formal position in the standards process. The IETF accepts over 1,000 individual drafts annually, and anyone may submit one.
The AIPREF working group already has two other drafts that have progressed further along the standards pathway: one establishing a vocabulary for AI-related preferences (the "vocabulary draft") and one detailing how those preferences could be communicated through the widely deployed robots.txt files (the "attachment draft"). In IETF terminology, these two drafts have been "adopted" by the AIPREF working group, indicating that participants achieved rough consensus to pursue these methods to accomplish the group's objectives.
The policy token draft lacks such consensus, being a single individual submission not yet reviewed by the group. Following discussion, the group may choose to adopt it, integrate its elements into the attachment draft, or reject it entirely.
2. A training preference exists, but search-specific preferences are absent, while industry attention has pivoted to inference.
Over the past two years, much of the AIPREF working group's effort has focused on defining new AI-related preferences in the vocabulary draft. These definitions matter significantly because they establish the categories for which websites can express preferences in their robots.txt files and through other technical channels.
Following extensive discussion, the group has made headway toward finalizing a definition of "AI training," though some wording questions persist. After even more—and more contentious—debate, a definition of what it means for website material to be used in "search" has also been (largely) provisionally settled. Both definitions received revisions and fresh language after the working group convened in person at the end of August. Heated discussions continue around a broader and less clearly defined category—variously termed "AI use," "inference," "grounding," and comparable applications distinct from model training.
The policy token draft specifically draws on the "AI training" definition from the vocabulary draft and proposes a way for sites to express a preference blocking all AI training across all crawlers by inserting a few lines into their existing robots.txt files. The attachment draft already includes a comparable mechanism, but the policy token draft's advantage lies in its immediate machine-readability by numerous already-deployed crawlers. For site owners burdened by maintaining an ever-expanding list of individual crawlers to disallow, this new mechanism offers the appeal of a single, straightforward way to block all crawlers from AI training.
What about search uses? And inference-time uses in general?
Notably absent from the policy token draft is an equivalent mechanism for search, particularly striking given that the two companies commanding nearly the entire web search market have already committed to supporting it. They could have sent a powerful signal by making it simple for sites to declare "search: yes; AI training: no." The search definition already exists in the vocabulary draft and could have been applied identically to the AI training definition. Publishers' lack of ability to express search-specific preferences has been the focus of regulatory scrutiny and action across numerous jurisdictions, making this gap especially conspicuous.
The proposal's emphasis on training also perpetuates the challenging distinctions that remain unresolved regarding inference-time uses broadly. Although the AIPREF working group has tested multiple formulations for an inference or "AI usage" category, agreement has proven elusive, even as commercial licensing negotiations have increasingly centered on inference and recurring revenue. Microsoft's statement revealed that the company will pair support for the policy token mechanism with its own proprietary grounding preference mechanism, underscoring the demand for preference signals extending beyond training.
3. Since the mechanism remains unstandardized, EU AI Act Code of Practice signatories are not yet obligated to recognize it.
Google's statement asserted that the proposed mechanism aligns with "the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act." However, the Code of Practice text itself specifies that signatories must follow only standardized crawler instructions:
Signatories commit … to employ web-crawlers that read and follow instructions expressed in accordance with the Robot Exclusion Protocol (robots.txt), as specified in the Internet Engineering Task Force (IETF) Request for Comments No. 9309, and any subsequent version of this Protocol for which the IETF demonstrates that it is technically feasible and implementable by AI providers and content providers, including rightsholders
EU Code of Practice for General Purpose AI Model Providers
As noted, the policy token draft having been released less than a week ago remains distant from formal standardization, extending RFC 9309, or establishing technical feasibility. While all parties presumably desire swift resolution, allowing interested stakeholders to examine and discuss the mechanism's design represents a vital component of the process that produces more robust standards, easier deployment, and (ideally) better alignment with website operators' needs.
Conclusion
A unified mechanism backed by the largest AI and search companies could meaningfully ease the load on publishers who now must handle crawler rules individually. Yet clarity about the announcement's scope remains essential: the preference applies solely to AI training, excluding search and other inference-time applications. As an individual draft, it has numerous steps remaining before formal standardization, during which its design may shift substantially. This process must be permitted to unfold before a handful of the world's largest corporations effectively set the rules the broader web must follow.
