{"id":22177,"date":"2025-06-29T05:05:56","date_gmt":"2025-06-29T05:05:56","guid":{"rendered":"https:\/\/youzum.net\/tencent-open-sources-hunyuan-a13b-a-13b-active-parameter-moe-model-with-dual-mode-reasoning-and-256k-context\/"},"modified":"2025-06-29T05:05:56","modified_gmt":"2025-06-29T05:05:56","slug":"tencent-open-sources-hunyuan-a13b-a-13b-active-parameter-moe-model-with-dual-mode-reasoning-and-256k-context","status":"publish","type":"post","link":"https:\/\/youzum.net\/fr\/tencent-open-sources-hunyuan-a13b-a-13b-active-parameter-moe-model-with-dual-mode-reasoning-and-256k-context\/","title":{"rendered":"Tencent Open Sources Hunyuan-A13B: A 13B Active Parameter MoE Model with Dual-Mode Reasoning and 256K Context"},"content":{"rendered":"

Tencent\u2019s Hunyuan team has introduced Hunyuan-A13B<\/strong>, a new open-source large language model<\/a> built on a sparse Mixture-of-Experts (MoE)<\/strong> architecture. While the model consists of 80 billion total parameters, only 13 billion are active during inference, offering a highly efficient balance between performance and computational cost. It supports Grouped Query Attention (GQA)<\/strong>, 256K context length<\/strong>, and a dual-mode reasoning framework<\/strong> that toggles between fast and slow thinking.<\/p>\n

Designed for efficient deployment and robust reasoning, Hunyuan-A13B achieves top-tier performance across agentic benchmarks including BFCL-v3<\/strong>, \u03c4-Bench<\/strong>, C3-Bench<\/strong>, and ComplexFuncBench<\/strong>, often outperforming larger models in tool-calling and long-context scenarios.<\/p>\n

Architecture: Sparse MoE with 13B Active Parameters<\/strong><\/h3>\n

At its core, Hunyuan-A13B follows a fine-grained MoE design comprising 1 shared expert<\/strong> and 64 non-shared experts<\/strong>, with 8 experts activated per forward pass<\/strong>. This architecture, backed by scaling experiments, ensures performance consistency while keeping inference costs low. The model includes 32 layers, uses SwiGLU<\/strong> activations, a vocabulary size of 128K, and integrates GQA for enhanced memory efficiency during long-context inference.<\/p>\n

The model\u2019s MoE setup is paired with an optimized training curriculum<\/strong>: a 20T-token pretraining phase, followed by fast annealing and long-context adaptation. This last phase scales the context window first to 32K and then to 256K tokens using NTK-aware positional encoding, ensuring stable performance at large sequence lengths.<\/p>\n

Dual-Mode Reasoning: Fast and Slow Thinking<\/strong><\/h3>\n

A standout feature of Hunyuan-A13B is its dual-mode Chain-of-Thought (CoT)<\/strong> capability. It supports both a low-latency fast-thinking<\/strong> mode for routine queries and a more elaborate slow-thinking<\/strong> mode for multi-step reasoning. These modes are controlled through a simple tag system: \/no think<\/code> for fast inference and \/think<\/code> for reflective reasoning. This flexibility allows users to adapt computational cost to task complexity.<\/p>\n

Post-Training: Reinforcement Learning with Task-Specific Reward Models<\/strong><\/h3>\n

The post-training pipeline of Hunyuan-A13B includes multi-stage supervised fine-tuning (SFT)<\/strong> and reinforcement learning (RL)<\/strong> across both reasoning-specific and general tasks. The RL stages incorporate outcome-based rewards<\/strong> and tool-specific feedback<\/strong>, including sandbox execution environments for code and rule-based checks for agents.<\/p>\n

In the agent training phase, the team synthesized diverse tool-use scenarios with planner, checker, and tool roles, generating over 20,000 format combinations<\/strong>. This reinforced Hunyuan-A13B\u2019s ability to execute real-world workflows such as spreadsheet processing, information search, and structured reasoning.<\/p>\n

\n
\"\"<\/figure>\n<\/div>\n

Evaluation: State-of-the-Art Agentic Performance<\/strong><\/h3>\n

Hunyuan-A13B shows strong benchmark results<\/strong> across diverse NLP tasks:<\/p>\n