Reuters and Time Adopt Bot-Blocking Whitelists to Rein in AI Crawlers

Why It Matters

By forcing AI crawlers to earn access, major publishers aim to recoup the value of premium content and set a monetization precedent for the rapidly expanding AI‑driven scraping economy.

Key Takeaways

• Reuters and Time now default‑block AI bots, allowing only whitelisted agents.
• Time’s whitelist includes ~70 bots, managed via ScalePost platform.
• Reuters uses robots.txt, monitoring tools, and licenses with Microsoft, Meta.
• Bot friction adds latency and cost, encouraging scrapers to negotiate licensing.
• Early results show no traffic drop and lower server costs for Reuters.

Pulse Analysis

The surge of generative‑AI applications has turned news articles into raw data for large language models, prompting publishers to confront a hidden scraper economy. Earlier this year, People Inc. and The Atlantic experimented with allow‑list models, but the scale of unauthorized crawling forced a more aggressive posture. By publicly publishing a whitelist, outlets signal that access must be earned, turning a technical control into a business negotiation lever.

Reuters and Time have operationalized this philosophy. Reuters embeds a restrictive robots.txt file, supplements it with server‑level monitoring, and maintains a vetted list of bots that receive permission—ranging from search engines to internal automation tools. Time, meanwhile, leverages ScalePost to automate whitelist management for about 70 approved agents, including major AI labs and social platforms. Both firms report that the added friction—typically a two‑second latency and the need for proxy services—discourages casual scrapers and nudges them toward formal licensing agreements, as evidenced by Reuters’ existing deals with Microsoft and Meta.

The broader implication is a potential industry standard for AI content licensing. As publishers demonstrate that blocking can be cost‑neutral or even cost‑saving, others may adopt similar tactics, prompting the development of clearer legal frameworks and commercial products around AI‑ready content. The SPUR Coalition’s recent expansion and the IAB Tech Lab’s bot‑blocking guide underscore a collective move toward protecting digital journalism’s economic moat while still enabling legitimate AI innovation.