The Role of AI in Health and Care Planning, Delivery, and Rapid Evaluation [Session 4: Plenary]

Why It Matters

The critique highlights real-world harm and wasted investment from unvalidated AI tools and makes the case that policymakers and health systems should prioritize system redesign, external validation and causal thinking to realize AI’s benefits.

Summary

In a plenary on AI in health, Dr. Jess Moley framed AI as data+algorithm+model and warned it often operates on a patient’s "data shadow" rather than the whole person. She argued AI is being used mainly to squeeze efficiency from existing processes—ambient scribes, waitlist modelling and individualized risk scores—rather than to redesign systems. Citing a systematic review of NHS risk tools, she said roughly one-third showed small benefit, one-third no effect and one-third worsened care, attributing failures to poor validation, context collapse and misunderstandings of causality. Moley urged a shift from individual-level predictive hype toward careful, system-level deployment and rigorous evaluation.