The scalar gate problem in delta-rule models<\/strong><\/h2>\nA recurrent linear attention layer stores a matrix state St<\/sub><\/code> and reads it with the query. DeltaNet adds an active edit by subtracting the value currently associated with the current key. It uses a scalar step size \u03b2t<\/sub><\/code> to control how much to overwrite. Mamba-2 adds a data-dependent scalar decay \u03b1t<\/sub><\/code> for global forgetting. Gated DeltaNet combined both operations, but both gates remained scalar per head.<\/p>\nKimi Delta Attention (KDA) refines the decay side. It replaces the scalar \u03b1t<\/sub><\/code> with a channel-wise vector. KDA still keeps a single scalar \u03b2t<\/sub><\/code> for the active edit. That scalar controls two different things at once. It decides how much old content to erase on the key side. It also decides how much new content to commit on the value side. These two decisions act on different axes of the state. Tying them together is a modeling restriction, not a property of the delta rule.<\/p>\n\n![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/05\/Screenshot-2026-05-24-at-12.26.27-AM-1.png\")
https:\/\/github.com\/NVlabs\/GatedDeltaNet-2\/blob\/main\/paper\/GDN2_paper.pdf<\/figcaption><\/figure>\n<\/div>\nGated Delta Rule-2: two gates instead of one<\/strong><\/h2>\nGated DeltaNet-2 separates the two decisions through Gated Delta Rule-2. It introduces a channel-wise erase gate bt<\/sub> \u2208 [0,1]d<\/sup>k<\/sub><\/code> on the key axis. It also introduces a channel-wise write gate wt<\/sub> \u2208 [0,1]d<\/sup>v<\/sub><\/code> on the value axis. Both gates are produced by sigmoid projections of the token representation. The update applies decay before the active edit.<\/p>\nWritten compactly, the recurrence is:<\/strong><\/p>\nSt<\/sub> = (I \u2212 kt<\/sub> (bt<\/sub> \u2299 kt<\/sub>)\u22a4<\/sup>) Dt<\/sub> St\u22121<\/sub> + kt<\/sub> (wt<\/sub> \u2299 vt<\/sub>)\u22a4<\/sup><\/strong><\/code><\/p>\nHere Dt<\/sub> = Diag(\u03b1t<\/sub>)<\/code> is the channel-wise decay carried over from KDA. The left factor of the erase matrix stays kt<\/sub><\/code>, preserving the delta-rule write direction. The right factor becomes bt<\/sub> \u2299 kt<\/sub><\/code>, making the read direction channel-selective. The write term kt<\/sub> zt<\/sub>\u22a4<\/sup><\/code> uses zt<\/sub> = wt<\/sub> \u2299 vt<\/sub><\/code>, making the value update channel-selective.<\/p>\nWhen both gates collapse to the same scalar \u03b2t<\/sub><\/code>, the update recovers KDA exactly. When the decay \u03b1t<\/sub><\/code> also collapses to a scalar, it recovers Gated DeltaNet. Both prior models are preserved as tied subspaces of the new update.<\/p>\nIn the fast-weight view, Gated Delta Rule-2 is one online gradient step on a local regression loss. The decayed state stays close to memory, while the residual edit uses gated read and gated write targets.<\/p>\n
Chunkwise training and gate-aware backward<\/strong><\/h2>\nThe recurrence admits a chunkwise WY form that matches the structure used by KDA. Cumulative channel-wise decay is absorbed into the two factors of each rank-one erase. The per-chunk update becomes a product of asymmetric matrices of the form I \u2212 k\u0304r<\/sub> \u0113r<\/sub>\u22a4<\/sup><\/code>. The implementation uses chunk size C = 64<\/code> with fused Triton kernels.<\/p>\nFor the backward pass, the scalar shortcut used by KDA no longer applies. The write side contains a different diagonal gate over value channels. The erase side contains a different diagonal gate over key channels. So the gate factors must appear inside the dot products that accumulate gradients. The paper derives this gate-aware vector-Jacobian product explicitly. On Hopper GPUs, the fused WY backward kernel is restricted to two and four warps to avoid a Triton WGMMA layout assertion.<\/p>\n
Block design and hybrid model<\/strong><\/h2>\nGated DeltaNet-2 is used as the recurrent token mixer in a standard Transformer-style block. Query and key paths use linear projection, short causal convolution, SiLU, and L2 normalization. The value path uses linear projection, short convolution, and SiLU. The decay \u03b1t<\/sub><\/code>, erase gate bt<\/sub><\/code>, and write gate wt<\/sub><\/code> come from separate linear branches. The recurrent output is RMS-normalized, multiplied by a SiLU output gate, and projected back.<\/p>\nA hybrid variant inserts Sliding-Window Attention (SWA) after the recurrent mixer. A repeated cell contains Gated DeltaNet-2, an MLP, SWA, and another MLP. SWA handles exact local interactions, while the recurrent mixer compresses long histories. The hybrid retains linear sequence scaling with a bounded attention cache.<\/p>\n
Results at 1.3B parameters<\/strong><\/h2>\nAll models are 1.3B parameters trained on 100B FineWeb-Edu tokens. Parameter count and recurrent state size are matched across models. The recurrent state holds 262,144 floats per layer per batch element. Training length is 4K tokens, and hybrid models use a 2K SWA window. The Mamba-3 MIMO baseline uses rank R = 4<\/code>.<\/p>\nOn language modeling and commonsense reasoning, Gated DeltaNet-2 has the best average in both settings. The recurrent model averages 53.11 across LAMBADA and the reasoning suite. That sits above Mamba-3 MIMO at 52.39 and KDA at 52.28. In the hybrid setting, Gated DeltaNet-2 averages 53.97 against Mamba-3 MIMO at 52.72. Since recurrent state size is matched, the gain points to the update rule, not more memory.<\/p>\n
The clearest gains appear on RULER long-context retrieval. In the recurrent setting, S-NIAH-2 at 4K rises from 89.0 (KDA) to 93.0. S-NIAH-3 at 2K jumps from 63.2 (KDA) to 89.8. MK-NIAH-1 at 4K climbs from 28.0 (KDA) to 37.8. <\/p>\n
On real-world retrieval (SWDE, SQuAD, FDA, TriviaQA, NQ, DROP), Gated DeltaNet-2 also leads both settings. The recurrent average is 29.88 and the hybrid average is 42.28.<\/p>\n
Marktechpost\u2019s Visual Explainer<\/strong><\/h2>\n\n\n\n <\/span>
\n Gated DeltaNet-2 \u00b7 Quickstart<\/span>\n <\/div>\n\n 01<\/span> \/ 08<\/span>\n <\/div>\n<\/div>\n\n\n <\/p>\n
St<\/sub><\/code> and reads it with the query. DeltaNet adds an active edit by subtracting the value currently associated with the current key. It uses a scalar step size \u03b2t<\/sub><\/code> to control how much to overwrite. Mamba-2 adds a data-dependent scalar decay \u03b1t<\/sub><\/code> for global forgetting. Gated DeltaNet combined both operations, but both gates remained scalar per head.<\/p>\nKimi Delta Attention (KDA) refines the decay side. It replaces the scalar \u03b1t<\/sub><\/code> with a channel-wise vector. KDA still keeps a single scalar \u03b2t<\/sub><\/code> for the active edit. That scalar controls two different things at once. It decides how much old content to erase on the key side. It also decides how much new content to commit on the value side. These two decisions act on different axes of the state. Tying them together is a modeling restriction, not a property of the delta rule.<\/p>\n\nhttps:\/\/github.com\/NVlabs\/GatedDeltaNet-2\/blob\/main\/paper\/GDN2_paper.pdf<\/figcaption><\/figure>\n<\/div>\nGated Delta Rule-2: two gates instead of one<\/strong><\/h2>\nGated DeltaNet-2 separates the two decisions through Gated Delta Rule-2. It introduces a channel-wise erase gate bt<\/sub> \u2208 [0,1]d<\/sup>k<\/sub><\/code> on the key axis. It also introduces a channel-wise write gate wt<\/sub> \u2208 [0,1]d<\/sup>v<\/sub><\/code> on the value axis. Both gates are produced by sigmoid projections of the token representation. The update applies decay before the active edit.<\/p>\nWritten compactly, the recurrence is:<\/strong><\/p>\nSt<\/sub> = (I \u2212 kt<\/sub> (bt<\/sub> \u2299 kt<\/sub>)\u22a4<\/sup>) Dt<\/sub> St\u22121<\/sub> + kt<\/sub> (wt<\/sub> \u2299 vt<\/sub>)\u22a4<\/sup><\/strong><\/code><\/p>\nHere Dt<\/sub> = Diag(\u03b1t<\/sub>)<\/code> is the channel-wise decay carried over from KDA. The left factor of the erase matrix stays kt<\/sub><\/code>, preserving the delta-rule write direction. The right factor becomes bt<\/sub> \u2299 kt<\/sub><\/code>, making the read direction channel-selective. The write term kt<\/sub> zt<\/sub>\u22a4<\/sup><\/code> uses zt<\/sub> = wt<\/sub> \u2299 vt<\/sub><\/code>, making the value update channel-selective.<\/p>\nWhen both gates collapse to the same scalar \u03b2t<\/sub><\/code>, the update recovers KDA exactly. When the decay \u03b1t<\/sub><\/code> also collapses to a scalar, it recovers Gated DeltaNet. Both prior models are preserved as tied subspaces of the new update.<\/p>\nIn the fast-weight view, Gated Delta Rule-2 is one online gradient step on a local regression loss. The decayed state stays close to memory, while the residual edit uses gated read and gated write targets.<\/p>\n
Chunkwise training and gate-aware backward<\/strong><\/h2>\nThe recurrence admits a chunkwise WY form that matches the structure used by KDA. Cumulative channel-wise decay is absorbed into the two factors of each rank-one erase. The per-chunk update becomes a product of asymmetric matrices of the form I \u2212 k\u0304r<\/sub> \u0113r<\/sub>\u22a4<\/sup><\/code>. The implementation uses chunk size C = 64<\/code> with fused Triton kernels.<\/p>\nFor the backward pass, the scalar shortcut used by KDA no longer applies. The write side contains a different diagonal gate over value channels. The erase side contains a different diagonal gate over key channels. So the gate factors must appear inside the dot products that accumulate gradients. The paper derives this gate-aware vector-Jacobian product explicitly. On Hopper GPUs, the fused WY backward kernel is restricted to two and four warps to avoid a Triton WGMMA layout assertion.<\/p>\n
Block design and hybrid model<\/strong><\/h2>\nGated DeltaNet-2 is used as the recurrent token mixer in a standard Transformer-style block. Query and key paths use linear projection, short causal convolution, SiLU, and L2 normalization. The value path uses linear projection, short convolution, and SiLU. The decay \u03b1t<\/sub><\/code>, erase gate bt<\/sub><\/code>, and write gate wt<\/sub><\/code> come from separate linear branches. The recurrent output is RMS-normalized, multiplied by a SiLU output gate, and projected back.<\/p>\nA hybrid variant inserts Sliding-Window Attention (SWA) after the recurrent mixer. A repeated cell contains Gated DeltaNet-2, an MLP, SWA, and another MLP. SWA handles exact local interactions, while the recurrent mixer compresses long histories. The hybrid retains linear sequence scaling with a bounded attention cache.<\/p>\n
Results at 1.3B parameters<\/strong><\/h2>\nAll models are 1.3B parameters trained on 100B FineWeb-Edu tokens. Parameter count and recurrent state size are matched across models. The recurrent state holds 262,144 floats per layer per batch element. Training length is 4K tokens, and hybrid models use a 2K SWA window. The Mamba-3 MIMO baseline uses rank R = 4<\/code>.<\/p>\nOn language modeling and commonsense reasoning, Gated DeltaNet-2 has the best average in both settings. The recurrent model averages 53.11 across LAMBADA and the reasoning suite. That sits above Mamba-3 MIMO at 52.39 and KDA at 52.28. In the hybrid setting, Gated DeltaNet-2 averages 53.97 against Mamba-3 MIMO at 52.72. Since recurrent state size is matched, the gain points to the update rule, not more memory.<\/p>\n
The clearest gains appear on RULER long-context retrieval. In the recurrent setting, S-NIAH-2 at 4K rises from 89.0 (KDA) to 93.0. S-NIAH-3 at 2K jumps from 63.2 (KDA) to 89.8. MK-NIAH-1 at 4K climbs from 28.0 (KDA) to 37.8. <\/p>\n
On real-world retrieval (SWDE, SQuAD, FDA, TriviaQA, NQ, DROP), Gated DeltaNet-2 also leads both settings. The recurrent average is 29.88 and the hybrid average is 42.28.<\/p>\n
Marktechpost\u2019s Visual Explainer<\/strong><\/h2>\n\n\n\n <\/span>
\n Gated DeltaNet-2 \u00b7 Quickstart<\/span>\n <\/div>\n\n 01<\/span> \/ 08<\/span>\n <\/div>\n<\/div>\n\n\n <\/p>\n