Google DeepMind Is Using Gemini to Train Agents Inside Goat Simulator 3

Why It Matters

SIMA 2 shows that large‑language models can endow embodied agents with instruction‑following and self‑learning capabilities across diverse virtual settings, a prerequisite for more adaptable, general‑purpose robotics. Its progress signals a shift from narrow game‑specific AI toward versatile agents that could accelerate development of real‑world autonomous systems.

Summary

Google DeepMind unveiled SIMA 2, a new video‑game‑playing agent that leverages the Gemini large‑language model to interpret text, voice or drawing commands and act on pixel‑level inputs across a range of 3D worlds, including Goat Simulator 3. Trained on human gameplay from eight commercial titles and three internal environments, the agent can self‑improve through trial‑and‑error, using Gemini‑generated hints when it fails. In tests on novel, AI‑generated worlds, SIMA 2 demonstrated the ability to navigate, use tools and follow open‑ended instructions without preset goals. The system still struggles with long‑term memory and complex multi‑step tasks, but marks a step toward agents that can transfer skills to real‑world robots.