Can AI Chatbots Reason Like Doctors?

Why It Matters

The findings suggest AI could soon augment diagnostic accuracy in emergency settings, but the lack of consistent evaluation standards and risks of misinformation mean regulators, providers, and investors must navigate both rapid innovation and patient safety concerns.

Summary

A study published in Science on April 30 found that OpenAI’s o1‑preview large language model outperformed two internal‑medicine physicians on clinical reasoning tasks using real emergency‑room records, achieving an exact or near‑exact diagnosis 82% of the time versus 79% and 70% for the doctors. The research highlights both the promise of LLMs as decision‑support tools for clinicians and the variability in performance metrics across studies, with other work showing high error rates and hallucinations in chatbot medical advice. OpenAI has already launched ChatGPT for Clinicians and ChatGPT for Healthcare, and experts stress the need for prospective trials, standardized evaluation methods, and careful integration into clinical workflows rather than viewing AI as a replacement for doctors.