What Are the Chances AI Will Give You Accurate Health Advice? 50/50, Says a New Study

Why It Matters

The unreliability of AI-generated health advice poses risks for patients seeking low‑cost alternatives, potentially leading to misdiagnoses or delayed care. As AI tools become more accessible, regulators and clinicians must address accuracy gaps to protect public health.

Key Takeaways

• AI health advice correct only about half the time, study finds
• 49.6% of AI responses were problematic, with 19.6% highly problematic
• Cancer queries performed better; nutrition and fitness yielded more errors
• One‑third of Americans used AI for health advice in 2023
• Grok model showed slightly higher error rate than peers

Pulse Analysis

The surge of consumer‑focused artificial‑intelligence tools has reshaped how people gather medical information. Platforms like ChatGPT and Gemini are now embedded in smartphones and browsers, offering instant answers to health queries that once required a clinician’s office visit. This convenience, however, masks a critical flaw: a recent BMJ Open study revealed that AI’s medical advice is correct only about half the time, a statistic that aligns with a KFF poll showing a third of Americans already rely on these bots for health guidance.

Delving into the study’s data, experts highlighted that nearly half of AI‑generated responses were problematic, with almost one‑fifth classified as highly problematic. Accuracy varied dramatically by topic—cancer‑related questions, which draw heavily from peer‑reviewed literature, saw relatively better performance, while nutrition and athletic performance queries suffered from misinformation proliferated by unverified online sources. The Grok model, in particular, exhibited a slightly higher error rate, underscoring that not all chatbots are created equal. These inconsistencies erode trust and raise concerns about patients making health decisions based on incomplete or erroneous information.

The implications extend beyond individual risk; they challenge healthcare systems, insurers, and policymakers to establish safeguards. Regulators may need to mandate transparency about AI training data and enforce rigorous validation before deployment in consumer health contexts. Clinicians should proactively discuss AI limitations with patients, guiding them toward evidence‑based resources. Ongoing research must refine model alignment with medical standards, ensuring that future AI assistants complement, rather than replace, professional medical judgment.