Measuring Progress Toward AGI: A Cognitive Framework

Why It Matters

The taxonomy provides the first systematic, human‑referenced metric suite for AGI progress, enabling the industry to track breakthroughs with comparable standards. Mobilizing the research community through a high‑stakes hackathon accelerates the creation of robust, shared benchmarks.

Key Takeaways

• DeepMind defines ten core cognitive abilities for AGI evaluation.
• Framework uses human baseline comparisons across perception to social cognition.
• Kaggle hackathon offers $200,000 to build missing benchmarks.
• Focus tracks: learning, metacognition, attention, executive functions, social cognition.
• Three-stage protocol aligns AI performance with human distribution.

Pulse Analysis

The quest for artificial general intelligence has long suffered from a lack of unified measurement tools. By borrowing decades of insight from cognitive science, DeepMind’s taxonomy translates abstract intelligence concepts into concrete, testable abilities. This interdisciplinary approach mirrors how psychologists assess human cognition, offering a familiar yardstick for AI researchers and investors alike. As the field matures, such a framework could become the de‑facto standard for reporting progress, much like ImageNet did for computer vision.

Central to the proposal is a three‑stage evaluation pipeline. First, AI models are challenged with a broad suite of tasks covering each of the ten identified abilities, using held‑out datasets to prevent overfitting. Second, a demographically diverse human cohort completes the same tasks, establishing performance distributions that serve as baselines. Finally, each model’s results are mapped onto these human distributions, revealing where systems excel, lag, or match human capability. This relative scoring method not only quantifies competence but also highlights specific cognitive gaps that require research focus.

To move from theory to practice, DeepMind has teamed with Kaggle to launch a $200,000 hackathon targeting the five abilities with the largest evaluation gaps—learning, metacognition, attention, executive functions, and social cognition. Participants will leverage Kaggle’s Community Benchmarks platform to design, test, and submit novel assessment tools. The competition, running from March 17 to April 16, promises substantial financial incentives and the chance to shape the metrics that will define the next era of AI development. By crowd‑sourcing these benchmarks, the initiative accelerates the creation of a shared, transparent yardstick that could steer both academic research and commercial investment toward truly generalizable intelligence.