The KV Cache Wars?

Key Takeaways

• KV cache memory grows linearly with context length.
• 128K tokens on 70B model need ~42 GB cache.
• Memory bandwidth limits decode throughput.
• Three solution families: eviction, compression, hierarchical memory.
• Prefill stage suffers from compute and memory spikes.

Summary

A quiet but critical battle is unfolding in agentic AI infrastructure over the key‑value (KV) cache. The KV cache, which stores key and value projections for every token, scales linearly with context length, layer count, batch size, and heads, consuming up to ~42 GB for a 128K‑token window on a 70‑billion‑parameter model. This memory pressure creates bottlenecks during both the prefill phase and the decode phase, throttling latency and throughput on typical 80 GB GPUs. Researchers and hardware vendors are tackling the issue through eviction‑based sparse attention, quantization‑driven compression, and hierarchical memory architectures.

Pulse Analysis

The key‑value cache sits at the heart of transformer inference, storing the projected keys and values for every token so they can be reused during autoregressive generation. While this eliminates the need to recompute these matrices on each step, the cache’s size expands linearly with the number of tokens, layers, batch size, and attention heads. On a 70‑billion‑parameter model with a 128 K token context, the KV cache alone can occupy roughly 42 GB of GPU memory, leaving little room for the model weights on an 80 GB card and creating severe memory‑bandwidth constraints during decoding.

To address these pressures, the research community has converged on three primary strategies. Eviction‑based approaches prune or sparsify the cache, discarding less relevant entries to cut compute cost during the prefill stage. Compression techniques, such as quantization and low‑rank factorization, shrink the stored vectors while preserving most of the attention fidelity. Meanwhile, hardware‑oriented solutions restructure where the cache lives, leveraging hierarchical memory tiers—high‑speed SRAM, DRAM, and even NVMe—to keep active context close to the compute cores and offload older data. Each method balances trade‑offs between latency, accuracy, and hardware complexity.

The outcome of this "KV cache war" will shape the next generation of agentic AI systems that require million‑token context windows and persistent memory across sessions. Companies that master efficient cache management can deliver faster, cheaper, and more capable assistants, while those that lag may face prohibitive infrastructure costs. As GPUs evolve with larger memory pools and specialized tensor caches, and as software frameworks adopt smarter attention kernels, the industry moves toward a future where massive context lengths become routine rather than exceptional, unlocking richer interactions and more sophisticated reasoning capabilities.