Nvidia and Its Partners' KV Cache Extenders

Why It Matters

By extending KV‑cache beyond GPU memory, Nvidia’s solution enables long‑context LLM inference at scale, reducing latency and GPU costs while boosting overall AI infrastructure efficiency.

Key Takeaways

• CMX adds G3.5 flash tier for KV‑cache offload.
• Nvidia claims up to 5× power efficiency, tokens‑per‑second.
• STX reference design integrates BlueField‑4 DPUs, Spectrum‑X.
• Major storage vendors announce CMX‑compatible servers.
• Local‑SSD KV‑cache approaches risk being bypassed.

Pulse Analysis

The rapid growth of large language models has exposed a fundamental bottleneck: the limited high‑bandwidth memory on GPUs cannot hold the ever‑expanding KV‑cache needed for long‑context inference. Nvidia’s CMX platform tackles this by treating NVMe flash as a first‑class tier—designated G3.5—within the KV‑cache hierarchy. By exposing the flash tier to the GPU’s address space and orchestrating data movement through the Dynamo framework and NIXL library, CMX reduces costly recomputation of keys and values, effectively turning storage latency into a manageable trade‑off for massive token windows.

STX builds on CMX by providing a turnkey, rack‑scale reference architecture that embeds BlueField‑4 DPUs, ConnectX‑9 SuperNICs, and Spectrum‑X Ethernet into a unified storage fabric. This design creates a direct GPU‑to‑storage data path, eliminating traditional CPU bottlenecks and enabling sub‑millisecond KV‑cache fetches from NVMe‑oF devices. The partnership ecosystem—spanning VAST Data, Dell, HPE, NetApp, and others—has already announced CMX‑ready servers, suggesting a rapid rollout of AI‑native storage solutions that can sustain multi‑turn, agentic workloads across large clusters.

For enterprises, the promise of up to five‑times higher token throughput and comparable gains in power efficiency translates into lower operational costs and the ability to run more sophisticated AI applications without over‑provisioning GPU fleets. However, adoption hinges on the maturity of the software stack, integration complexity, and the willingness of data‑center operators to invest in specialized hardware. Competitors offering CXL‑based memory expansion or alternative KV‑cache offload methods may challenge Nvidia’s dominance, but the early momentum of the STX reference design positions Nvidia as a pivotal player in the next generation of AI inference infrastructure.