What Is A Website Crawler? 12 Bot Management Strategies

Summary

Ilya Grigorik explains that over half of web traffic now comes from bots, dividing them into good (search), grey (AI training/fetchers) and bad (malicious scrapers) and urging e‑commerce teams to treat this "post‑human" reality as a core business concern. He highlights the rise of "zero‑click" answer engines, the resulting dark funnel where traditional analytics miss bot visits, and the need for distinct strategies for indexing bots versus price‑checking fetchers. The episode outlines a 12‑step bot management playbook—from discovery via sitemaps and IndexNow to rendering challenges with JavaScript and proper HTTP status codes—while recommending schema, server‑side rendering, and behavioral security to stay visible to helpful bots and block harmful ones.