Why Physical AI Needed a Completely New Data Stack

Rerun.ai has built a purpose‑made logging and visualization stack that targets the unique demands of robotics and spatial computing. By abandoning traditional object‑oriented schemas and adopting an entity‑component system, the platform can ingest multimodal sensor streams—LiDAR point clouds, video, joint states, and arbitrary tensors—without rigid pre‑definition. An in‑memory database powers instant, high‑throughput visual debugging, a capability that most ML ops tools lack. This design mirrors the simplicity of Weights & Biases APIs while delivering the speed and flexibility required to troubleshoot complex embodied AI systems.

The conversation highlighted why robot manipulation has accelerated in the past year. Imitation learning, where humans teleoperate robots and the system records full state trajectories, now pairs with reinforcement learning to refine policies beyond the demonstrated examples. End‑to‑end transformer models and larger datasets have finally generalized to messy tasks such as folding laundry, a problem once deemed impossible. These algorithmic gains, combined with cheaper compute and richer sensor suites, are turning research prototypes into repeatable, robust behaviors that can be deployed at scale.

Despite the hype, production robotics still lags behind lab demos. Benchmarks remain scarce, forcing teams to co‑design data pipelines and hardware together, as Rerun’s customers do. Early adopters in manufacturing are deploying learning‑based pick‑and‑place robots in tens‑to‑hundreds units, but widespread scaling is pending reliable evaluation frameworks. The industry’s next step is a unified data stack that bridges fast logging, systematic benchmarking, and seamless model iteration—exactly the niche Rerun aims to fill. As open‑source models mature, practical, cost‑effective robots are poised to move from showcase to everyday utility.