{"id":93270,"date":"2026-05-27T17:12:21","date_gmt":"2026-05-27T17:12:21","guid":{"rendered":"https:\/\/youzum.net\/meet-eagle-3-1-the-speculative-decoding-algorithm-that-fixes-attention-drift-in-llm-inference\/"},"modified":"2026-05-27T17:12:21","modified_gmt":"2026-05-27T17:12:21","slug":"meet-eagle-3-1-the-speculative-decoding-algorithm-that-fixes-attention-drift-in-llm-inference","status":"publish","type":"post","link":"https:\/\/youzum.net\/zh\/meet-eagle-3-1-the-speculative-decoding-algorithm-that-fixes-attention-drift-in-llm-inference\/","title":{"rendered":"Meet EAGLE 3.1: The Speculative Decoding Algorithm That Fixes Attention Drift in LLM Inference"},"content":{"rendered":"

Speculative decoding is a technique for speeding up large language model inference. A small, fast draft model proposes several tokens. The large target model verifies them in parallel. If accepted, inference is faster. If rejected, the system falls back gracefully.<\/p>\n

EAGLE Team, vLLM Team, and TorchSpec Team has launched the EAGLE series including EAGLE 1, EAGLE 2, and EAGLE 3 has become one of the most widely adopted and practically deployed families of speculative decoding algorithms across both research and production systems. Today, that family gets a targeted reliability upgrade with introduction of EAGLE 3.1<\/a>. <\/p>\n

What was Going Wrong<\/strong><\/h2>\n

While speculative decoding performs well in controlled settings, performance often degrades under different chat templates, long-context inputs, or out-of-distribution system prompts. <\/p>\n

The EAGLE team traced this fragility to a phenomenon called attention drift<\/a><\/strong> as speculation depth increases, the drafter gradually shifts attention away from sink tokens and toward its own generated tokens. <\/p>\n

In simpler terms: the drafter is a small model that predicts future tokens. As speculation gets deeper, it starts attending to its own prior outputs instead of the original context. This degrades acceptance length and output stability.<\/p>\n

Two underlying issues were identified. First, the fused input representation becomes increasingly imbalanced as higher-layer hidden states dominate the drafter input. Second, hidden-state magnitude grows across speculation steps due to the unnormalized residual path. Together, these effects make the drafter progressively less stable at deeper speculation depths.<\/p>\n

Two Architectural Fixes in EAGLE 3.1<\/strong><\/h2>\n

To address attention drift, EAGLE 3.1 comes with two key architectural improvements: FC normalization after each target hidden state and before the FC layer, and feeding post-norm hidden states into the next decoding step.<\/p>\n

FC normalization stabilizes the hidden states that the drafter receives from the target model. Without it, hidden-state magnitude grows across steps, making the drafter increasingly unreliable. Applying normalization at each step keeps the inputs bounded.<\/p>\n

The post-norm design makes the method behave more like recursively invoking the drafter across decoding steps, rather than simply appending additional layers to the target model. <\/p>\n

\n
\"https:\/\/vllm.ai\/blog\/2026-05-26-eagle-3-1\"
https:\/\/vllm.ai\/blog\/2026-05-26-eagle-3-1<\/figcaption><\/figure>\n<\/div>\n

What These Fixes Deliver<\/strong><\/h2>\n

Compared with EAGLE 3, EAGLE 3.1 demonstrates: better training-time to inference-time extrapolation, stronger long-context robustness, higher resilience to chat template and system prompt variation, and more stable acceptance length across diverse serving environments. <\/p>\n

In long-context workloads, EAGLE 3.1 achieves up to 2\u00d7 longer acceptance length compared with EAGLE 3. <\/p>\n

Training Infrastructure: TorchSpec<\/strong><\/h2>\n

TorchSpec now provides efficient training support for EAGLE 3.1 and future speculative decoding algorithms. By lowering training overhead and simplifying experimentation workflows, TorchSpec helps accelerate iteration and exploration for next-generation speculative decoding research and deployment. <\/p>\n

Based on TorchSpec and vLLM, the research team also trained and open-sourced an EAGLE 3.1 draft model for Kimi K2.6, available on HuggingFace<\/a>. The model serves as an example of deploying EAGLE 3.1 with TorchSpec training and vLLM serving support on a real-world serving model<\/p>\n

vLLM Integration: Config-Driven and Backward-Compatible<\/strong><\/h2>\n

EAGLE 3.1 lands in vLLM as a config-driven extension of the existing EAGLE 3 implementation. The integration includes FC normalization support, post-norm hidden-state feedback, and removal of hardcoded assumptions around target hidden states. <\/p>\n

Backward compatibility with existing EAGLE 3 checkpoints is fully preserved. EAGLE 3.1 draft models can be plugged directly through the same speculative-decoding code path. <\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
vllm serve nvidia\/Kimi-K2.6-NVFP4 \n  --trust-remote-code \n  --tensor-parallel-size 4 \n  --tool-call-parser kimi_k2 \n  --enable-auto-tool-choice \n  --reasoning-parser kimi_k2 \n  --attention-backend tokenspeed_mla \n  --speculative-config '{\"model\":\"lightseekorg\/kimi-k2.6-eagle3.1-mla\",\"method\":\"eagle3\",\"num_speculative_tokens\":3}' \n  --language-model-only<\/code><\/pre>\n<\/div>\n<\/div>\n
Benchmark Results on Kimi K2.6<\/strong><\/h2>\n
The research team benchmarked the Kimi K2.6 EAGLE 3.1 draft model on Kimi-K2.6-NVFP4 with vLLM (TP=4, GB200, non-disagg) on the SPEED-Bench coding dataset. EAGLE 3.1 delivers 2.03\u00d7 higher per-user output throughput at concurrency 1. The speedup stays meaningful as concurrency scales: 1.71\u00d7 at C=4 and 1.66\u00d7 at C=16. <\/p>\n
Marktechpost\u2019s Visual Explainer<\/strong><\/h2>\n
\n
<\/div>\n
\n
<\/p>\n
\n01 \/ 07<\/span>\n
\nvLLM \u00b7 May 26, 2026<\/span>\n
Meet EAGLE 3.1<\/h1>\n
<\/span>
\nThe EAGLE team, vLLM team, and TorchSpec team jointly released EAGLE 3.1 \u2014 a targeted fix for speculative decoding instability in production LLM serving.<\/span><\/p>\n
\n#speculative-decoding<\/span>
\n#vLLM<\/span>
\n#LLM inference<\/span>
\n#performance<\/span>\n<\/div>\n<\/div>\n<\/div>\n
<\/p>\n
\n02 \/ 07<\/span>\n
\nBackground<\/span>\n
What is Speculative Decoding?<\/h2>\n
<\/span>
\nA technique for speeding up LLM inference using two models working together.<\/span><\/p>\n