AI News and Headlines

Differential Transformer V2

The Differential Transformer V2 (DIFF V2) introduces a differential attention operation that doubles query heads while keeping key‑value heads unchanged, eliminating the need for custom attention kernels. By projecting a per‑token, per‑head λ and applying a sigmoid‑scaled subtraction, DIFF V2 removes the per‑head RMSNorm used in V1, stabilizing gradients and reducing activation outliers. The design preserves decoding speed comparable to standard Transformers and saves roughly 25% of attention‑module parameters. Early large‑scale pre‑training experiments show a modest LM loss improvement and fewer gradient spikes versus baseline models.