Half the Web Is Bots — Inside Fastly's Threat Report and the New AI Crawler Problem

Why It Matters

Understanding and controlling bot traffic is essential for preserving bandwidth, protecting content revenue, and preventing AI‑driven data theft, making advanced bot‑management a strategic priority for digital businesses.

Key Takeaways

• Bots generate 49% of web traffic, 99% are unwanted.
• Unwanted bots often hit cached content, creating hidden business risks.
• Distinguishing good AI crawlers from malicious bots requires advanced visibility tools.
• Fastly’s bot management combines WAF and analytics to granularly block threats.
• AI fetcher traffic is rising, monetizing content without creator consent.

Summary

The interview with Fastly CISO Marshall Irwin centers on the company’s latest threat report, which reveals that nearly half of all web requests now originate from bots, and almost all of that traffic is unwanted. Irwin explains how Fastly’s position as a major CDN gives it unique visibility into both legitimate and malicious traffic, allowing the firm to publish detailed quarterly analyses of bot trends. Key data points include 49% of requests being bot‑driven, with 99% classified as harmful, and roughly 47% of that unwanted traffic targeting cached content rather than origin servers. This cache‑level activity creates a stealthy business risk, as bots harvest and monetize content without generating value for site owners. Irwin also highlights the surge in AI‑related traffic—both crawlers that train models and fetchers that augment real‑time AI services. Notable quotes underscore the challenge: Irwin notes that “visibility is really key” and that “you need to look at bot intent, not just bot identity.” He stresses that traditional binary block/allow approaches are insufficient; granular controls and advanced analytics are required to differentiate between search engine bots, AI crawlers, and malicious actors. The implications are clear: organizations must adopt sophisticated bot‑management solutions, such as Fastly’s integrated WAF and bot visibility platform, to protect bandwidth, preserve content value, and mitigate emerging AI‑driven threats. Failure to do so risks both operational costs and loss of intellectual property as AI agents increasingly scrape the web.

Original Description

In this episode of TechStrong TV, Alan Shimel sits down with Marshall Erwin, Chief Information Security Officer at Fastly, for a sharp look at what's really moving across the modern web — and how AI is reshaping the bot landscape from the inside out.

Fastly sits in a unique vantage point: it powers a large share of the world's web traffic — from major media outlets like The Guardian, to GitHub, to leading streaming services — and protects those customers with DDoS, WAF and bot management on top of its CDN. Marshall walks through Fastly's newest threat report, which finds that roughly 49% of all requests Fastly customers see are now bot-driven, and 99% of that bot traffic is unwanted. He also unpacks the more hidden problem: about 47% of unwanted bot traffic is hitting cached content, where the cost shows up not as origin load but as quiet, ongoing business risk as content is vacuumed up and monetized elsewhere.

Alan and Marshall dig into:

- Why bot management has shifted from a narrow security problem (account takeover) to a board-level strategic issue

- AI crawlers vs. AI fetchers — what each one actually does, and why fetcher traffic is the fastest-growing category

- The cached-content scraping problem and why CDN-level visibility is critical

- Why blunt yes/no blocking is no longer viable — and the case for intent-aware bot management

- Why agentic AI traffic is largely traffic site owners actually want, and how to differentiate it

- The role of WAF + bot management in giving sites the granular controls they need

- How AI itself is becoming central to detecting and responding to AI-driven traffic

- Where to find Fastly's threat report and dig into the data yourself

A must-watch for CISOs, platform owners, web ops teams, SEO/AEO leads and anyone trying to understand what AI is doing to web traffic — and what to do about it.

Chapters:

00:00 Introduction — Marshall Erwin joins the show

00:30 Marshall's path from the US intel community to Mozilla CSO to Fastly CISO

02:30 What Fastly actually does and where it sits in the CDN landscape

04:30 The Fastly Threat Report — 49% of web traffic is bots, 99% unwanted

06:30 From narrow security problem to board-level strategic challenge

08:00 The case for intent-based bot management and visibility

10:00 The hidden problem — 47% of bad bot traffic hits cached content

12:00 What organizations should actually do — and why most don't see the problem

14:00 AI crawlers vs. AI fetchers — training data vs. real-time productized AI

15:30 Agentic traffic — the bots websites will actually want

16:30 Where to get the Fastly threat report

17:00 Closing thoughts

Guest: Marshall Erwin, Chief Information Security Officer, Fastly — https://www.fastly.com

Host: Alan Shimel, TechStrong Group

Subscribe to TechStrong TV for more conversations with the leaders shaping web security, DevOps, AI and the future of the internet.

#Fastly #BotManagement #AICrawlers #WebSecurity #CDN