Know What's Happening in AI

Are We Testing AI’s Intelligence the Wrong Way?

At NeurIPS, AI expert Melanie Mitchell argued current AI evaluation relies on benchmarks that fail to capture true cognition. She advocated borrowing experimental methods from developmental and comparative psychology, such as controlled variations and failure analysis, to probe non‑verbal intelligences like babies and animals. Mitchell highlighted the need for replication and skepticism in AI research, warning that current metrics can mislead about real‑world capabilities. She also questioned the nebulous AGI definition and urged more rigorous, psychology‑inspired testing.