2,000 Robots Walk Into a Shop: Simulated A/B Testing (2026) – Shopify

Why It Matters

SimGym turns costly, slow A/B testing into a minutes‑scale, low‑cost service, empowering small merchants to iterate quickly and improve conversion rates. The breakthrough showcases how tailored LLM inference pipelines can unlock new business models for e‑commerce platforms.

Key Takeaways

• SimGym runs 2,000 concurrent browser bots for A/B testing
• Uses open‑source 120B MoE model with FlashInfer optimizations
• Blackwell GPUs deliver 5× speedup over H200
• MIG partitioning cuts latency 20% and boosts throughput
• Speculative decoding adds 6% throughput, ready for production

Pulse Analysis

The rise of AI‑driven simulation platforms like Shopify’s SimGym marks a shift in how e‑commerce firms validate storefront changes. Traditional A/B testing relies on real customer traffic, often taking weeks to reach statistical significance—or failing entirely for low‑volume merchants. By deploying hundreds of autonomous agents that mimic diverse shopper personas, SimGym generates synthetic traffic that interacts with live page renders, producing actionable conversion data in minutes. This approach not only accelerates product iteration cycles but also democratizes testing, giving small retailers the same data fidelity previously reserved for high‑traffic brands.

At the heart of SimGym’s performance is a custom inference stack built around an open‑source 120‑billion‑parameter Mixture‑of‑Experts model. Engineers applied MXFP4 weight quantization, FP8 attention caching, and bespoke FlashInfer kernels to reduce memory‑bound bottlenecks on NVIDIA’s Blackwell GPUs. The result is a five‑fold token‑throughput increase compared with earlier H200 deployments, slashing per‑session latency and driving a 10‑12 % rise in daily merchant runs. Complementary techniques—MIG GPU partitioning and speculative decoding with the EAGLE‑3 head—further trim response times and boost throughput without sacrificing output quality.

The broader implication for the industry is clear: tailored LLM serving pipelines can transform latency‑heavy, sequential workloads into scalable services. As SimGym demonstrates, combining hardware‑level optimizations with model‑specific tricks like prompt caching and guided JSON generation creates a cost‑effective, high‑throughput engine for complex agentic tasks. Companies that invest in such end‑to‑end stacks will gain a competitive edge, delivering rapid, data‑driven insights to merchants and unlocking new revenue opportunities across the digital commerce ecosystem.