{"id":89858,"date":"2026-05-12T16:25:39","date_gmt":"2026-05-12T16:25:39","guid":{"rendered":"https:\/\/youzum.net\/tilde-research-introduces-aurora-a-leverage-aware-optimizer-that-fixes-a-hidden-neuron-death-problem-in-muon\/"},"modified":"2026-05-12T16:25:39","modified_gmt":"2026-05-12T16:25:39","slug":"tilde-research-introduces-aurora-a-leverage-aware-optimizer-that-fixes-a-hidden-neuron-death-problem-in-muon","status":"publish","type":"post","link":"https:\/\/youzum.net\/th\/tilde-research-introduces-aurora-a-leverage-aware-optimizer-that-fixes-a-hidden-neuron-death-problem-in-muon\/","title":{"rendered":"Tilde Research Introduces Aurora: A Leverage-Aware Optimizer That Fixes a Hidden Neuron Death Problem in Muon"},"content":{"rendered":"

Researchers at Tilde Research have released Aurora<\/strong>, a new optimizer for training neural networks that addresses a structural flaw in the widely-used Muon optimizer. The flaw quietly kills off a significant fraction of MLP neurons during training and keeps them permanently dead. Aurora<\/strong> comes with a 1.1B parameter pretraining experiment, a new state-of-the-art result on the modded-nanoGPT speedrun benchmark, and open codes.<\/p>\n

What is Muon?<\/strong><\/h2>\n

To understand Aurora, it helps to first understand Muon. The Muon optimizer attracted attention in the ML community after outperforming AdamW in wall-clock time to convergence<\/strong> on the nanoGPT speedrun competition \u2014 a community benchmark that measures how fast you can train a GPT-style model to a target validation loss. Since then, Muon has been adopted in frontier-scale model training by several research groups.<\/p>\n

Muon\u2019s key algorithmic step is computing the polar factor<\/strong> of the gradient matrix. For a gradient matrix G<\/strong> with thin Singular Value Decomposition (SVD) G = U\u03a3V\u1d40<\/strong>, Muon computes polar(G) = UV\u1d40<\/code>, which is the closest semi-orthogonal matrix to G<\/strong> in the Frobenius norm. This orthogonalized gradient is then used to update the weights: W \u2190 W \u2212 \u03b7 UV\u1d40<\/strong> for a learning rate \u03b7. The use of matmul-only iterative algorithms to compute the polar factor is what makes Muon practical at scale.<\/p>\n

The NorMuon Puzzle: Row Normalization Helps, But Why?<\/strong><\/h2>\n

Before Aurora, NorMuon led the modded-nanoGPT speedrun. It introduced a row-normalization step\u2014similar to Adam\u2019s per-parameter scaling\u2014that adjusted the polar factor by its inverse RMS norm. While this often pulls the update away from a strictly orthogonal gradient, NorMuon still yields impressive results. The Tilde team set out to understand exactly what gap in Muon\u2019s formulation NorMuon was addressing. <\/p>\n

The Core Problem: Row-Norm Anisotropy and Neuron Death in Tall Matrices<\/strong><\/h2>\n

The research team discovered that the Muon optimizer unintentionally \u201ckills\u201d a large portion of neurons in tall weight matrices<\/strong>, such as those found in SwiGLU-based MLP layers. Because it is mathematically impossible for these specific matrix shapes to stay perfectly orthogonal while keeping row updates even, the optimizer ends up giving massive updates to some neurons while virtually ignoring others. This results in a \u201cdeath spiral\u201d where under-performing neurons receive less signal over time, eventually becoming permanently inactive.<\/p>\n

The research study revealed that by the 500th training step, more than one in four neurons are effectively dead. This isn\u2019t just a local issue; the lack of activity in these neurons starves subsequent layers of necessary data, spreading the inefficiency throughout the model. Aurora<\/strong> solves this by using a new mathematical approach that enforces uniform updates across all neurons without sacrificing the benefits of orthogonalization.<\/p>\n

The Intermediate Step: U-NorMuon<\/strong><\/h2>\n

Before arriving at Aurora, the research introduces an intermediate fix called U-NorMuon<\/strong>. The key observation is that NorMuon normalizes each row to unit norm (norm = 1), but this is actually the wrong target for a tall matrix. For a column-orthogonal tall matrix, the mathematically correct average row norm is \u221a(n\/m), not 1. U-NorMuon corrects this by normalizing tall matrix rows to have norm \u221a(n\/m) instead of 1.<\/p>\n

In experiments at 340M scale, U-NorMuon outperforms both Muon and standard NorMuon and completely eliminates the neuron death phenomenon \u2014 leverage scores become approximately isotropic throughout training. Crucially, U-NorMuon propagates this benefit to layers it doesn\u2019t directly touch: keeping up\/gate rows alive ensures isotropic gradient flow into the down-projection, stabilizing its column leverage without any direct intervention.<\/p>\n

However, U-NorMuon still has a problem: it forcefully overrides the polar factor with uniform row norms, sacrificing polar factor precision, which is both theoretically undesirable and empirically costly in the Muon framework (the paper shows that Muon achieves monotonically lower loss with more precise orthogonalization). This is the motivation for Aurora.<\/p>\n

Aurora: Steepest Descent Under Two Joint Constraints<\/strong><\/h2>\n

Aurora reformulates the update-selection problem from scratch. Rather than running orthogonalization and then patching it with row normalization, Aurora asks: what is the optimal update under the joint<\/strong> constraint of left semi-orthogonality and uniform row norms?<\/p>\n

Formally, for tall matrices, Aurora solves:<\/p>\n

U\u2217=argUmax\u200bTr(G\u22a4U)s.t.U\u22a4U=In\u200b,\u2225Ui:\u200b\u22252=mn\u200b\u2200iU \u2217 =arg U max \u200b Tr(G \u22a4 U)s.t.U \u22a4 U=I n \u200b ,\u2225U i: \u200b \u2225 2 = m n \u200b \u2200i<\/strong><\/p>\n

The research shows that these two constraints together force all singular values of U to exactly equal 1. This means the joint constraint still produces a valid left semi-orthogonal update, not a compromised one. This is the key insight that separates Aurora from NorMuon and U-NorMuon: it achieves row-norm uniformity and orthogonality simultaneously rather than trading one off against the other.<\/p>\n

The research also provides two algorithmic implementations of Aurora\u2019s solution. The Riemannian Aurora<\/strong> uses a gradient projection approach restricted to the joint Stiefel\/equal-row-leverage manifold. The vanilla Aurora<\/strong> is a simpler, more practical implementation. Both are open-sourced. For non-tall (wide and square) matrices, row-norm uniformity is already implied by orthogonality, so Aurora leaves those parameters unchanged.<\/p>\n

Results<\/strong><\/h2>\n

Aurora was used to train a 1.1B model that achieves 100x data efficiency on open-source internet data and outperforms larger models on general evals like HellaSwag. At 1B scale, Aurora achieves large gains over both Muon and NorMuon. On the modded-nanoGPT optimization speedrun, Aurora\u2019s submitted run outperforms the prior state-of-the-art (which was NorMuon). Untuned Aurora carries only a 6% compute overhead over traditional Muon and is designed as a drop-in replacement.<\/p>\n

The research team also found that Aurora\u2019s performance gains scale with MLP width, suggesting it is particularly effective for networks with large MLP expansion factors \u2014 which is consistent with the neuron death hypothesis, since wider MLPs have more tall matrices and more opportunity for leverage anisotropy to compound.<\/p>\n

Key Takeaways<\/strong><\/h2>\n