AI Dev 26 X SF | Andi Partovi: Why Every Agent Needs a Simulation Sandbox

Why It Matters

Using simulation sandboxes reduces the risk of costly operational, legal or customer-facing failures by catching unpredictable behaviors and policy violations before agents act on real systems. It provides a scalable, repeatable framework for evaluating and improving autonomous agents that traditional testing cannot deliver.

Summary

Various AI CTO Andi Partovi argued that builders of autonomous, action-based agents must use realistic simulation sandboxes to test and harden systems before production. He explained agents are nondeterministic, interactive, and operate in partially observable environments, so traditional static test sets and predefined labels fail to capture real-world behavior. Simulations emulate users, tools and services at scale, allow repeated runs to surface edge cases and failure modes, and enable post-run labeling and iterative improvement. Partovi presented simulation as the practical way to validate agent safety, policy compliance and robustness prior to live deployment.

Original Description

AI agents fail in unpredictable ways that traditional testing can't catch — hallucinations, wrong tool calls, policy violations, and more. Teams only discover these failures after users hit them in production.

A simulation sandbox gives you a controlled environment with realistic users, tools, and workflows where you can run hundreds of scenarios against your agent before it ships, catching edge cases and adversarial inputs that would be impossible to test manually.

This talk by Veris AI's Andi Partovi covers why simulation-driven development is becoming essential infrastructure for any team building production AI agents, and how it closes the gap between "works in demos" and "works at scale."