THE DECODER

AI Models Confidently Describe Images They Never Saw, and Benchmarks Fail to Catch It

A new study reveals that leading multimodal AI models—including GPT‑5 series, Gemini 3 Pro, and Claude Opus 4.5—confidently generate visual descriptions and medical diagnoses despite never receiving an image, achieving 60‑90% correctness in a text‑only benchmark called Phantom‑0. When tested on established visual‑understanding benchmarks, these models attain 70‑80% of their full scores using only textual cues, with medical tests reaching up to 99% of image‑based performance. A 3‑billion‑parameter text‑only model fine‑tuned on a chest‑X‑ray dataset outperformed all frontier multimodal systems and radiologists by over 10%. The researchers propose the B‑Clean framework to strip away questions solvable without images, exposing inflated rankings and reshaping model evaluation.