{"id":105186,"date":"2026-07-18T19:29:58","date_gmt":"2026-07-18T19:29:58","guid":{"rendered":"https:\/\/youzum.net\/sakana-ais-error-diffusion-trains-dale-compliant-dual-stream-networks-reaching-96-7-mnist-and-61-7-cifar-10-without-backpropagation\/"},"modified":"2026-07-18T19:29:58","modified_gmt":"2026-07-18T19:29:58","slug":"sakana-ais-error-diffusion-trains-dale-compliant-dual-stream-networks-reaching-96-7-mnist-and-61-7-cifar-10-without-backpropagation","status":"publish","type":"post","link":"https:\/\/youzum.net\/es\/sakana-ais-error-diffusion-trains-dale-compliant-dual-stream-networks-reaching-96-7-mnist-and-61-7-cifar-10-without-backpropagation\/","title":{"rendered":"Sakana AI\u2019s Error Diffusion Trains Dale-Compliant Dual-Stream Networks, Reaching 96.7% MNIST and 61.7% CIFAR-10 Without Backpropagation"},"content":{"rendered":"

Backpropagation dominates deep learning, yet it uses a mechanism the brain likely cannot. Specifically, the backward pass needs exact transposes of forward weight matrices. This is the weight transport problem<\/em>. Sakana AI\u2019s new paper, Diffusing Blame<\/em><\/a>, confronts this constraint directly. The research team trains networks that obey Dale\u2019s principle<\/strong> while avoiding weight transport entirely. <\/p>\n

What is Error Diffusion?<\/strong><\/h2>\n

Error Diffusion (ED) is a local learning rule, first proposed by Kaneko (2000). Each weight update depends on three signals only. These are presynaptic activity, a postsynaptic activation derivative, and a single global error sign. Consequently, ED never transports transposed forward weights or uses random feedback matrices. That locality makes ED naturally compatible with Dale\u2019s principle. However, prior work demonstrated ED only on binary classification and MNIST.<\/p>\n

The Dual-Stream Architecture<\/strong><\/h2>\n

To satisfy that constraint, the research team split each layer into two streams. One stream is excitatory (p<\/strong>), and the other is inhibitory (n<\/strong>). The forward pass computes excitatory-minus-inhibitory preactivations for each stream:<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
p_i = \u03c6_i( +p_{i-1} Wpp  \u2212 n_{i-1} Wnp  + bp )\nn_i = \u03c6_i( +n_{i-1} Wnn  \u2212 p_{i-1} Wpn  + bn )<\/code><\/pre>\n<\/div>\n<\/div>\n
Here, all four weight matrices stay non-negative element-wise. The biases bp<\/sub><\/code> and bn<\/sub><\/code> are the exception, since they need not be non-negative. Moreover, the negation signs before Wnp<\/sub><\/code> and Wpn<\/sub><\/code> are structural, not learned. Therefore cross-stream connections remain inhibitory while all learnable weights stay non-negative. This design needs four weight sub-matrices per layer. As a result, it uses roughly 4\u00d7 more parameters than a single-stream network. For the same architecture, that is \u223c32M versus \u223c8M for DFA.<\/p>\n
Modulo Error Routing<\/strong><\/h2>\n
With that architecture in place, the main extension is modulo error routing<\/strong>. This lifts Error Diffusion (ED) beyond binary classification. For hidden unit i<\/em>, the research team define the routing r(i) = i mod C<\/code>. Here, C<\/em> is the output dimension. That unit then learns from the routed error component. In short, each hidden unit is assigned one fixed output channel. Unlike DFA, whose feedback matrices are random, ED uses this structured correspondence.<\/p>\n
Three Classification Innovations<\/strong><\/h2>\n
Building on that routing, the research team adds three fixes for multi-class classification:<\/strong><\/p>\n
    \n
  1. Layer-specific sigmoid widths<\/strong> use \u03c6i<\/sub>(z) = 1\/(1 + e\u22122z\/\u03b1i<\/sub><\/sup>)<\/code>. Since the sigmoid derivative directly gates the error signal, attenuation is severe. In fact, post-hoc analysis reveals a 25\u00d7 decay from the output to the first hidden layer. Wider sigmoids keep derivatives larger, preventing premature saturation. The team sets \u03b1 = 3.0 for CIFAR-10 convolutional layers and \u03b1 = 6.0 for fully connected layers.<\/li>\n
  2. Batch-centered class error<\/strong> subtracts the per-class mini-batch mean. This makes the one-vs-all error zero-mean across the batch for every class. It thereby reduces persistent suppression caused by the 9:1 target imbalance.<\/li>\n
  3. Asymmetric initialization<\/strong> scales excitatory weights by 1.5\u00d7 and inhibitory weights by 0.5\u00d7. That gives an expected E\/I scale ratio of 3:1, while the output layer stays symmetric.<\/li>\n<\/ol>\n
    <\/p>\n
    \n<\/div>\n
    \n
    Performance<\/strong><\/h2>\n<\/p>
    With all three innovations, Error Diffusion (ED) reaches 96.7% on MNIST and 61.7% on CIFAR-10. In contrast, seed ED without them collapses to 50.4% and 11.6%. DFA scores higher on both tasks but violates Dale\u2019s principle, using \u223c2.84M negative weights. Notably, this is the first time ED has trained convolutional networks. Previously, Fujita (2026) reached \u223c55.2% on CIFAR-10 using a flattened MLP. Even so, 61.7% remains far from standard gradient-based methods.<\/p>\n
    \n\n\n\n\n\n\n\n
    Method<\/th>\nMNIST<\/th>\nCIFAR-10<\/th>\nDale-compliant<\/th>\nNotes<\/th>\n<\/tr>\n<\/thead>\n
    Proposed ED<\/strong><\/td>\n96.7%<\/td>\n61.7%<\/td>\nYes<\/td>\nAll weights non-negative; first ED on CNNs<\/td>\n<\/tr>\n
    Seed ED<\/td>\n50.4%<\/td>\n11.6%<\/td>\nYes<\/td>\nNo innovations; \u03b1 = 1.0, raw error, symmetric init<\/td>\n<\/tr>\n
    DFA<\/td>\n97.6%<\/td>\n69.1%<\/td>\nNo<\/td>\nRandom feedback; \u223c2.84M negative weights<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n
    The Ablation Reversal<\/strong><\/h2>\n
    Interestingly, the innovations\u2019 importance flips between tasks. On MNIST, removing layer-specific widths is catastrophic (\u221271.4 pp), collapsing accuracy toward chance. Batch-centering barely matters there (\u22120.3 pp). On CIFAR-10, however, the order reverses. Removing batch-centered error becomes the largest drop (\u221247.9 pp), collapsing four of five seeds. This reversal exposes task-dependent credit-assignment bottlenecks invisible to single-benchmark evaluation.<\/p>\n
    Error Diffusion in Reinforcement Learning<\/strong><\/h2>\n
    Beyond classification, the research team integrate ED with Proximal Policy Optimization (PPO). They call the result ED-PPO<\/strong> and test it on Brax locomotion and Craftax. Here, policy-output error is routed to hidden units by output channel. For the scalar value network, the error is broadcast to all units. Importantly, ED-PPO drops the three classification innovations entirely. Across five seeds, ED-PPO beats BP-PPO on HalfCheetah (5494 vs 3520; p < 0.001) and matches DFA-PPO. On Ant, it stays on par with both PPO variants. On Craftax, meanwhile, DFA-PPO is the weakest method (19.8 vs BP-PPO 27.0). Thus random feedback that suffices for classification can fail on open-ended RL.<\/p>\n
    Use Cases and Examples<\/strong><\/h2>\n
    Three settings make this concrete<\/strong>:<\/p>\n