The Free and Open Web Is Under Attack at the IETF

Key Takeaways

• IETF's AI Preferences group proposes robots.txt signals to block AI scraping.
• Web Bot Auth aims to cryptographically authenticate bots, enabling selective access.
• Potential licensing could shut out researchers, archivists, and small startups.
• EFF and allies have already blocked several restrictive IETF proposals.

Pulse Analysis

Automated crawling has become the backbone of modern information ecosystems. Journalists use scrapers to uncover hidden stories, archivists preserve the web’s fleeting content, and consumers rely on price‑comparison bots to find better deals. As artificial‑intelligence models grow hungry for massive datasets, publishers fear bandwidth strain and revenue loss, prompting a push for technical safeguards. This tension has spilled into standards‑making bodies, where the balance between protecting site performance and preserving open access is being debated.

The IETF’s AI Preferences working group is drafting a mechanism that would let sites broadcast “preference signals” via robots.txt, indicating they do not wish their data harvested for AI training or search. If jurisdictions treat those signals as legally binding, a simple text file could become a de‑facto gatekeeper. Simultaneously, the Web Bot Auth group is exploring cryptographic identification of bots, allowing sites to create whitelist‑only ecosystems. While the intent—to curb abusive traffic—is legitimate, the technology could be weaponized to charge licensing fees or block competitors, effectively monetizing the right to read publicly available content.

For the broader internet economy, the stakes are high. Restrictive standards could cripple independent research, limit accessibility tools for disabled users, and stifle innovation among startups that cannot afford licensing deals. The Electronic Frontier Foundation and a coalition of open‑internet stakeholders have already mobilized to defeat earlier proposals, emphasizing that neutrality in core protocols is essential for a vibrant digital commons. A balanced approach that mitigates abusive bots without eroding the open web will be crucial as AI continues to reshape how data is consumed and repurposed.