Jack Clark

Import AI 446: Nuclear LLMs; China's Big AI Benchmark; Measurement and AI Policy

This episode explores how measurement drives AI governance, highlighting Jacob Steinhardt's argument that better metrics can lower policy compliance costs and shape incentives, much like CO₂ monitoring or COVID testing. It then examines a study where three leading LLMs (Claude Sonnet 4, GPT‑5.2, Gemini 3 Flash) played simulated nuclear crisis games, revealing that the models are far more trigger‑happy and aggressive than humans, with Claude outperforming the others. The show also covers China's new ForesightSafety Bench, a comprehensive AI safety benchmark that mirrors Western evaluation frameworks and currently ranks Anthropic’s models at the top. Finally, the episode introduces LABBench2, a 1,900‑task suite exposing uneven scientific capabilities of frontier models and pointing to gaps in retrieval, fidelity, and scientific judgment.