Lowering the Cost of Intelligence With NVIDIA's Ian Buck - Ep. 284

The episode opens with a clear definition of Mixture of Experts (MoE) and why it matters for modern AI. By partitioning a massive neural network into dozens of specialized "experts" and routing each token to only the most relevant ones, MoE models can achieve the same or higher intelligence scores while activating a fraction of the total parameters. This selective activation translates into dramatically lower token‑compute costs—often ten times cheaper than dense models—making large‑scale inference economically viable for enterprises.

Ian Buck highlights the DeepSeek moment as the catalyst that brought MoE into the mainstream. DeepSeek’s open‑source model demonstrated that a 256‑expert per‑layer architecture could outperform closed‑source rivals on benchmark leaderboards, proving that the router‑expert‑combiner pipeline works at scale. The conversation explains how the router learns to dispatch queries to the right experts without hard‑coded domain labels, and how multiple experts per layer can be consulted in parallel. This architectural shift has sparked a wave of new open models that all rely on MoE to push intelligence scores upward while keeping token costs down.

The final segment ties model innovation to NVIDIA’s hardware roadmap. Advances such as HBM memory, NVLink, and the newer MVLink interconnect enable dozens of GPUs to act as a single, high‑bandwidth engine, allowing each expert to reside on its own GPU slice. This co‑design delivers X‑factor performance gains—sometimes 15× faster inference—while only modestly increasing per‑GPU cost. The result is a dramatic reduction in cost‑per‑token, empowering developers to deploy ever larger, smarter models without prohibitive expense. Buck concludes that continued GPU scaling and interconnect improvements will keep the MoE ecosystem both cutting‑edge and cost‑effective for the next generation of AI applications.