Sebastian Raschka

Hyper‑connections Residuals Boost Transformer Stability and Performance

Efficiency and performance tweaks in the transformer architecture usually focus(ed) on the normalization, attention, and FFN modules. For instance: Normalization: LayerNorm -> RMSNorm -> Dynamic TanH Attention: Grouped-query attention, sliding window, multi-head latent attention, sparse attention FFN: GeLU -> SiLU, SiLU -> SwiGLU, Mixture of Experts. Well, I just saw the New Year’s gift from DeepSeek, which includes some improvements to the residual path. In short, it’s built on the hyper-connections (HC) approach, which generalizes the regular (identity) residual connection into a learned one by widening the residual stream via multiple parallel ones and allowing information to mix across those parallel layers. They then take the HC idea a step further and propose mHC, which constrains the residual mixing to lie on a structured, norm-preserving manifold. They found that this "m"-modification improves training stability. This adds a small amount of overhead, but they get much better training stability and convergence. Arxiv link to the paper: https://lnkd.in/gSSgev3r