Differential Transformer V2

Why It Matters

DIFF V2 boosts LLM efficiency and training stability without sacrificing speed, offering a practical upgrade for production‑scale transformers.

Key Takeaways

• Doubles query heads without extra KV heads
• Removes per-head RMSNorm, stabilizing gradients
• Achieves decoding speed comparable to standard Transformers
• Saves ~25% attention module parameters
• Improves language modeling loss by 0.02–0.03

Pulse Analysis

Differential Transformer V2 builds on the original DIFF concept by re‑engineering the attention block to use twice as many query heads while sharing the same key‑value cache. This architectural tweak aligns query, key, and value dimensions, allowing the use of off‑the‑shelf FlashAttention kernels and avoiding the custom kernels that slowed DIFF V1. The per‑token, per‑head λ parameter, passed through a sigmoid, scales the subtraction between paired attention heads, effectively controlling the context RMS and eliminating the need for a separate RMSNorm layer.

From a performance standpoint, DIFF V2 raises the arithmetic intensity of the attention module during decoding, a phase that is typically memory‑bound for large language models. Because KV heads remain unchanged, the value cache is loaded only once, preserving the throughput of standard Transformers on H‑series and B‑series GPUs. The result is decoding latency that matches baseline models while the output projection matrix stays the same size, delivering roughly a 25% reduction in attention‑module parameters that can be reallocated to other network components.

Empirical results from trillion‑token pre‑training runs, including a 30‑billion‑parameter mixture‑of‑experts model, indicate that DIFF V2 consistently lowers language‑modeling loss by 0.02–0.03 points and curtails gradient spikes even at aggressive learning rates (6e‑4 to 1e‑3). The removal of RMSNorm also mitigates activation outliers, enhancing numerical stability. These gains make DIFF V2 an attractive drop‑in upgrade for organizations seeking faster, more stable LLM training and inference without extensive code rewrites.