Amazon's Anti-Benchmark AI Bet

Summary

In this episode, host interviews Amazon’s AI chief Rohit Prasad, who argues that the AI community should stop obsessing over benchmark leaderboards and focus on real‑world utility, noting that current evals are noisy and incomparable. He explains Amazon’s contrarian approach, emphasizing consistent training data and held‑out evaluations as the true measure of progress, and hints at upcoming announcements at AWS re:Invent that showcase practical advancements. The conversation also touches on industry reactions, including OpenAI’s “code red” and Anthropic’s new Claude model, underscoring a shift toward performance that matters beyond scores.