Why Are Large Language Models so Terrible at Video Games?

Why It Matters

The inability of LLMs to master video games signals limits in using game‑like simulations for general AI training, cautioning investors and developers about over‑reliance on LLMs for interactive tasks.

Key Takeaways

• LLMs excel at coding but fail video game tasks
• Game diversity outpaces LLM training data
• Benchmarks show no progress across varied games
• Spatial reasoning remains a weak spot for LLMs
• Simulation‑based AI may need non‑LLM approaches

Pulse Analysis

The rapid ascent of large language models in code generation has reshaped software development, but their triumphs mask a blind spot: interactive, real‑time decision making. Video games combine visual perception, spatial reasoning, and rapidly changing inputs—domains absent from the text‑heavy corpora that train LLMs. As a result, models that can compile flawless code stumble when asked to navigate a platformer or strategize in a real‑time strategy title, exposing a fundamental mismatch between language‑centric training and embodied intelligence.

Benchmarking efforts illustrate this gap. The General Video Game AI competition, which introduced fresh titles each year, saw agents improve on some games while regressing on others, a trend that persisted after LLMs entered the arena. Without exposure to game‑specific visual data and with limited spatial reasoning capabilities, LLMs perform worse than even basic search algorithms. Moreover, the scarcity of high‑quality, game‑specific datasets—unlike the abundant coding tutorials and test suites—prevents models from learning the nuanced cause‑effect loops that games demand.

For industry stakeholders, the takeaway is clear: relying solely on LLMs to power simulation‑based AI training may be premature. Companies like Nvidia and Google must complement language models with reinforcement‑learning agents, multimodal perception modules, and domain‑specific data pipelines. By acknowledging the distinct skill sets required for interactive environments, firms can better allocate R&D resources, avoid overhyped expectations, and develop more robust AI systems capable of both reasoning and action.