Why Vision Language Models Ignore What They See with Munawar Hayat - #758

In this episode, Munawar Hayat from Qualcomm AI Research explains why modern vision‑language models (VLMs) often ignore visual input, leading to hallucinations and poor physics‑aware generation. He highlights that when a language model is fused with a vision encoder, the massive textual pre‑training dwarfs the visual signal, causing the system to answer from parametric memory rather than the image. The discussion references recent NeurIPS papers that systematically analyze this failure, showing tangled similarity distributions between image and text embeddings and demonstrating that many benchmarks can be solved without actually looking at the picture.

Hayat’s team tackles the problem by redesigning the transformer architecture: instead of concatenating vision and text tokens at the input, they insert cross‑attention modules every few layers and add an auxiliary loss that forces attention onto relevant visual regions. Offline segmentation masks—derived from models like SAM—guide the loss, ensuring the model focuses on the correct object while keeping inference speed unchanged. This hierarchical injection reduces the quadratic self‑attention cost (from (M+N)^2 to N^2+MN), enabling longer contexts and more efficient training for multimodal generative AI on mobile hardware.

The conversation also critiques existing benchmarks, noting that datasets such as ScienceQA or AI2D allow language‑only shortcuts, while newer vision‑centric suites like CVBench demand true visual grounding. By emphasizing spatial correspondence and object‑centric queries, these benchmarks expose the vision‑ignoring behavior and drive research toward robust visual understanding. Hayat predicts that as Qualcomm scales these techniques, we’ll see more reliable VLMs on edge devices, better physics‑aware image synthesis, and fewer hallucinations—advancing both practical applications and fundamental AI research.