Tilde Research Introduces Aurora: A Leverage-Aware Optimizer That Fixes a Hidden Neuron Death Problem in Muon

Why It Matters

Aurora restores full model capacity in large‑scale training, delivering faster convergence and higher performance for AI developers, which can lower compute costs and accelerate innovation in the competitive LLM market.

Key Takeaways

• Muon optimizer kills >25% of MLP neurons early training
• Aurora enforces orthogonal updates and uniform row norms jointly
• Aurora improves data efficiency 100× on 1.1B model
• Only 6% compute overhead versus Muon, drop‑in replacement
• Performance gains grow with wider MLP expansion factors

Pulse Analysis

The optimizer landscape for large language models has been dominated by AdamW and its variants, but the recent Muon optimizer gained traction after beating AdamW in wall‑clock time on the nanoGPT speedrun. Researchers soon discovered a subtle structural issue: in tall weight matrices typical of SwiGLU‑based MLP layers, Muon’s polar factor update creates row‑norm anisotropy, causing more than 25% of neurons to die by the 500th step. This hidden neuron death reduces effective model capacity and propagates inefficiencies throughout deeper layers, limiting the scalability of otherwise fast training pipelines.

Aurora tackles the problem at its mathematical core. Instead of applying orthogonalization first and then patching it with row‑norm scaling, Aurora solves a joint optimization problem that simultaneously enforces left‑semi‑orthogonal updates and equal row norms. The resulting update matrix retains perfect singular values of one, preserving the benefits of Muon’s polar factor while guaranteeing uniform neuron activation. Two implementations—Riemannian Aurora, which projects gradients onto the Stiefel/equal‑row‑leverage manifold, and a simpler vanilla version—are open‑sourced, making the approach accessible for both research and production environments.

Empirical results validate Aurora’s design. A 1.1 billion‑parameter model trained with Aurora achieved a 100× data‑efficiency boost on open‑source internet data and outperformed larger competitors on benchmarks like HellaSwag. On the modded‑nanoGPT speedrun, Aurora set a new state‑of‑the‑art, surpassing the previous NorMuon record while incurring only a 6% compute overhead. Because the gains scale with MLP width, developers of next‑generation LLMs can expect even larger improvements, positioning Aurora as a practical, cost‑effective upgrade for high‑performance AI training pipelines.