Beyond the AI Hype in Health Care Services

The discourse surrounding artificial intelligence in health care has become saturated with breathless predictions and vague proclamations. Not only does this create distrust, but it also just makes reading about AI quite boring!

Industry white papers promise that AI will “revolutionize” medicine; conference keynotes describe a future where algorithms diagnose disease with superhuman accuracy; and venture capital flows freely toward startups claiming to transform clinical workflows. Yet for all the enthusiasm, the specifics remain elusive. What exactly does AI do when deployed in a real clinical environment? Does it actually improve patient outcomes, or does it simply produce outputs that look impressive on paper? These questions matter enormously, because the distance between a promising proof-of-concept and a genuinely useful clinical tool is often far greater than advocates acknowledge.

The problem is not that AI lacks potential in health care. The use of machine learning, deep learning, and other statistical learning methods clearly has enormous potential to inform diagnoses, clinical operations, population-level interventions, and treatment.

The problem is that most popular accounts skip over the messy, ambiguous reality of implementation. They focus on benchmark performance rather than bedside performance, on what a model can do in isolation rather than what happens when a clinician actually uses it. The result is a literature and a public conversation that is long on promise and short on evidence about what works, what doesn’t, and why.

The evidence for AI is on a long-journey ahead. One study with positive results does not necessarily translate to all studies and vice-versa.

Here are a few recent studies that help provide some thought and evidence along the road to AI in health care.

Where AI Meets Medicine

AI applications in health care now span a wide range of tasks. Computer vision algorithms read medical images detecting diabetic retinopathy, flagging suspicious mammograms, identifying pathology slides. Natural language processing systems extract information from clinical notes, automate discharge summaries, and classify adverse event reports. Predictive models estimate patient deterioration risk, forecast readmission probability, and stratify populations for targeted interventions. On the operational side, AI tools optimize operating room scheduling, manage supply chains, and route patients through emergency departments. When multiple variables need to be considered for making optimal decisions, computers can help inform and automate management decisions.

These applications vary dramatically in their maturity, evidence base, and proximity to direct patient impact. Image-based diagnostic tools have accumulated the most rigorous evidence through prospective trials, though even here, real-world performance often falls short of the controlled conditions under which algorithms were developed and validated. Meanwhile, tools designed to support complex clinical reasoning (e.g., the kind of nuanced, multi-variable decision-making that characterizes most medical encounters) remain far less proven.

The Rise of Large Language Models in Clinical Settings

The emergence of large language models (LLMs) has introduced a qualitatively different kind of AI into health care. Unlike narrow models trained for a single classification task, LLMs process and generate natural language, enabling them to engage with the unstructured text that dominates clinical documentation: patient histories, progress notes, medication lists, and clinical reasoning. This flexibility has generated considerable interest about their potential as clinical decision support tools, documentation assistants, and even diagnostic aids.

LLMs have demonstrated impressive performance on medical licensing examinations and clinical vignettes, sometimes matching or exceeding physician-level accuracy on standardized tests. But standardized tests are not clinical practice. The real question is whether LLM capabilities translate into measurable improvements when deployed alongside clinicians caring for actual patients. Three recent studies offer substantive, methodologically distinct answers to this question, and their findings collectively paint a picture that is more nuanced and more cautionary than the prevailing narrative suggests.

Study 1: LLMs as Medication Safety Co-Pilots

Ong et al. (2025), published in Cell Reports Medicine, evaluated LLMs as clinical decision support systems for medication safety across 16 medical and surgical specialties. The researchers developed and validated five LLM models using a retrieval-augmented generation (RAG) framework and tested them against 91 prescribing error scenarios embedded within 40 complex clinical vignettes at Singapore General Hospital. The study compared three implementation modes: LLM alone, pharmacist alone, and a co-pilot mode where pharmacists used the LLM as a decision aid.

The co-pilot approach performed best, achieving 61% accuracy in identifying drug-related problems—a 32.6% improvement over pharmacists working alone (46% accuracy). The LLM alone achieved 51% accuracy. For errors capable of causing serious harm, the co-pilot mode detected two-thirds of problems, representing a 1.5-fold improvement over pharmacists alone. Claude 3.5 Sonnet emerged as the best-performing model and demonstrated high reproducibility across repeated queries.

However, the study also revealed important limitations. RAG augmentation did not consistently improve performance over the native model, in fact, Claude 3.5 Sonnet performed slightly worse with RAG than without it. Performance declined in the co-pilot mode for one critical category: inappropriate dosage regimens, where accuracy dropped from 52% to 43%. The authors attributed this partly to geographic variability in dosing guidelines and the model’s reliance on pretraining data that may not reflect local institutional standards.

The differences between modes were not statistically significant overall (p = 0.38), and accuracy showed no correlation with case complexity. The study was also limited to five DRP categories, used fictional scenarios, and tested only a single prompt strategy, all factors constraining generalizability.

Study 2: Automated Auditing of Clinical Trial Reporting Quality

One of the most important and immediate use-cases for LLMs in health care exists in research. There is a lot of information published each day and the ability for a human to keep up with all of it is labor intensive and nearly impossible. Translating research into practice is a critical function in medicine and health care services.

Srinivasan et al. (2025), published in JAMA Network Open, applied a zero-shot LLM pipeline to a fundamentally different problem: assessing the reporting quality of randomized clinical trials against the CONSORT checklist. Rather than supporting individual clinical decisions, this study used LLMs to audit the transparency and completeness of published research at scale (i.e., a task previously limited by the labor-intensive nature of manual review).

The researchers screened over 53,000 open-access RCTs from PubMed and analyzed 21,041 articles spanning 1966 to 2024 across 30 biomedical disciplines. GPT-4o-mini achieved a macro F1 score of 0.86 on a benchmark dataset and agreed with human expert judgments 91.7% of the time across 70 validated articles. The model demonstrated high reliability, producing identical results across repeated runs.

The substantive findings were striking. Overall CONSORT compliance improved from 27.3% in the pre-1990 era to 57.0% after 2010, but critical methodological elements remained poorly reported even in recent publications: only 16.1% of trials described allocation concealment mechanisms, just 1.6% discussed external validity, and 2.2% provided protocol access information.

Compliance varied widely by discipline, from 35.2% in pharmacology to 63.4% in urology. Notably, neither FDA regulation, data monitoring committees, nor safety reporting requirements were associated with meaningfully better compliance.

This study demonstrates a genuinely productive use case for LLMs, one where the technology’s ability to process large volumes of text at scale addresses a real bottleneck in research quality assurance. The limitations are real but bounded: the analysis captured only the presence of reporting elements rather than their quality, and four CONSORT items were excluded because the model struggled to distinguish between events that did not occur and events that were simply not reported.

Study 3: LLM Decision Support in a Real Clinical Setting

The most methodologically ambitious of the three studies is Abaluck et al. (2026), an NBER working paper that deployed LLM decision support in two outpatient clinics in Kano, Nigeria. Community Health Extension Workers drafted care plans that were then revised after receiving LLM feedback. Uniquely, every patient was also independently examined by an on-site physician who developed their own care plan and then performed blinded evaluations of both the unassisted and LLM-assisted health worker notes.

Health workers responded enthusiastically to the LLM: they changed prescriptions for 54% of patients, modified diagnoses for 41%, and altered test orders for 33%. In exit surveys, 95% reported the LLM improved documentation and patient care, and 100% said they would use it again. Retrospective academic reviewers also rated LLM-assisted plans more favorably, finding significant reductions in errors and harm ratings.

Yet the physicians who had actually examined the same patients found little to no improvement. LLM assistance produced no statistically significant reduction in diagnostic errors, incorrect medications, or overall patient harm. Laboratory testing for malaria, UTI, and anemia revealed mixed effects: the LLM reduced unnecessary malaria testing but increased unnecessary testing for UTI and anemia, with almost no improvement in detecting positive cases. Treatment misallocation showed no significant change.

The mechanism analysis was particularly revealing. The LLM made an average of 3.75 recommendations per patient, but only 53% of these would have moved health worker plans closer to physician plans, meaning nearly half would have made things worse.

Health workers followed about one-third of recommendations and were only marginally better than chance at distinguishing helpful from unhelpful advice. Even when the LLM was given the physician’s more detailed patient observations rather than the health worker’s notes, it continued to generate more than one counterproductive recommendation per patient.

What These Studies Tell Us Together

Read collectively, these three studies offer a corrective to both uncritical enthusiasm and reflexive skepticism about AI in health care. LLMs can meaningfully augment clinician performance in structured tasks like medication chart review, where they serve as a second set of eyes catching errors that humans miss. They can scale quality assurance processes—like CONSORT compliance auditing—that were previously impractical and labor intensive. But when deployed for complex clinical reasoning with real patients, the gap between what looks good on paper and what actually improves care becomes starkly apparent.

In my opinion, the net-net here is that the details continue to matter.

The divergence between retrospective evaluation and prospective, physician-grounded assessment in the Abaluck study is perhaps the most important finding across all three papers. It suggests that the dominant method of evaluating clinical AI, comparing model outputs against records after the fact, may systematically overestimate real-world benefit. The “human in the loop” does not reliably filter out bad AI advice, and LLMs generate a substantial volume of recommendations that would make care worse, not better. At least according to that one study.

These are not reasons to abandon AI in health care. They are reasons to demand specificity: about what task the AI is performing, how it is being evaluated, and whether the evidence comes from controlled benchmarks or from the messy reality of clinical practice.

The evidence for AI is on a long-journey ahead. One study with positive results does not necessarily translate to all studies and vice-versa.

Here are a few recent studies that help provide some thought and evidence along the road to AI in health care.

Healthcare Pulse

Beyond the AI Hype in Health Care Services

Why It Matters

Ask Pulse AI: