AI Videos

Scaling Intelligence Through the Memory Hierarchy with Solidigm

Kapil Kirkra, senior principal engineer at Solidigm, argued that scaling AI intelligence requires a third, often overlooked axis: memory capacity. While larger models and more compute dominate headlines, the talk demonstrated how the memory hierarchy—from high‑bandwidth HBM to NVMe SSD tiers—directly influences model performance and quality. Using a single RTX 6000 Pro GPU with 96 GB HBM, Kirkra showed that when the KV cache fits within HBM, the system achieves 29 requests per second (system‑one recall). Expanding the working set beyond the 22 GB usable cache forces recomputation, dropping throughput to 2.68 rps (system‑two). Similarly, on the AIM 2024 math benchmark, a 32‑billion‑parameter model scored 7% on the first run but rose to 83% when given sufficient token budget, illustrating how extra capacity fuels tenacity and higher accuracy. Key examples included a jump from 67% to 80% and then 83% on the math test by allocating parallel chain‑of‑thought instances within the 22 GB HBM limit, and the stark finding that any truncation of reasoning tokens resulted in a 0% score. These data points underscore that memory capacity—not just compute—determines whether a model can retain context, reason deeply, and deliver reliable outputs. The implication for enterprises is clear: investing in tiered memory architectures and expanding KV cache capacity can dramatically improve AI service throughput, reduce latency, and boost the quality of complex inference tasks. Companies that overlook this lever risk slower, less accurate AI deployments and higher operational costs.