{"id":45445,"date":"2025-10-19T07:09:54","date_gmt":"2025-10-19T07:09:54","guid":{"rendered":"https:\/\/youzum.net\/microsoft-ai-proposes-bitnet-distillation-bitdistill-a-lightweight-pipeline-that-delivers-up-to-10x-memory-savings-and-about-2-65x-cpu-speedup\/"},"modified":"2025-10-19T07:09:54","modified_gmt":"2025-10-19T07:09:54","slug":"microsoft-ai-proposes-bitnet-distillation-bitdistill-a-lightweight-pipeline-that-delivers-up-to-10x-memory-savings-and-about-2-65x-cpu-speedup","status":"publish","type":"post","link":"https:\/\/youzum.net\/it\/microsoft-ai-proposes-bitnet-distillation-bitdistill-a-lightweight-pipeline-that-delivers-up-to-10x-memory-savings-and-about-2-65x-cpu-speedup\/","title":{"rendered":"Microsoft AI Proposes BitNet Distillation (BitDistill): A Lightweight Pipeline that Delivers up to 10x Memory Savings and about 2.65x CPU Speedup"},"content":{"rendered":"

Microsoft Research proposes BitNet Distillation<\/strong>, a pipeline that converts existing full precision LLMs into 1.58 bit<\/strong> BitNet students for specific tasks, while keeping accuracy close to the FP16 teacher and improving CPU efficiency. The method combines SubLN based architectural refinement<\/strong>, continued pre training<\/strong>, and dual signal distillation<\/strong> from logits and multi head attention relations. Reported results show up to 10\u00d7 memory savings<\/strong> and about 2.65\u00d7 faster CPU inference<\/strong>, with task metrics comparable to FP16 across multiple sizes. <\/p>\n

What BitNet Distillation changes?<\/strong><\/h3>\n

The community already showed that BitNet b1.58<\/strong> can match full precision quality when trained from scratch, but converting a pretrained FP16 model directly to 1.58 bit<\/strong> often loses accuracy, and the gap grows as model size increases. BitNet Distillation targets this conversion problem for practical downstream deployment. It is designed to preserve accuracy while delivering CPU friendly ternary weights with INT8 activations.<\/p>\n

Stage 1: Modeling refinement with SubLN<\/strong><\/h3>\n

Low bit models suffer from large activation variance. The research team inserts SubLN<\/strong> normalization inside each Transformer block, specifically before the output projection of the MHSA module<\/strong> and before the output projection of the FFN<\/strong>. This stabilizes hidden state scales that flow into quantized projections, which improves optimization and convergence once weights are ternary. The training loss curves in the analysis section support this design.<\/p>\n

Stage 2: Continued pre training to adapt weight distributions<\/strong><\/h3>\n

Direct task fine tuning at 1.58 bit<\/strong> gives the student only a small number of task tokens, which is not enough to reshape the FP16 weight distribution for ternary constraints. BitNet Distillation performs a short continued pre training<\/strong> on a general corpus, the research team uses 10B tokens<\/strong> from the FALCON corpus, to push weights toward BitNet like distributions. The visualization shows the mass concentrating near transition boundaries, which makes small gradients flip weights among [-1, 0, 1] during downstream task training. This improves learning capacity without a full pretraining run.<\/p>\n

Stage 3: Distillation based fine tuning with two signals<\/strong><\/h3>\n

The student learns from the FP16 teacher using logits distillation<\/strong> and multi head self attention relation distillation<\/strong>. The logits path uses temperature softened KL between teacher and student token distributions. The attention path follows the MiniLM and MiniLMv2<\/strong> formulations, which transfer relations among Q, K, V without requiring the same number of heads, and let you choose a single layer to distill. Ablations show that combining both signals works best, and that selecting one well chosen layer preserves flexibility.<\/p>\n

Understanding the results<\/strong><\/h3>\n

The research team evaluates classification, MNLI, QNLI, SST 2, and summarization on CNN\/DailyMail dataset. It compares three settings, FP16 task fine tuning, direct 1.58 bit task fine tuning, and BitNet Distillation. Figure 1<\/strong> shows that BitNet Distillation matches FP16 accuracy for Qwen3 backbones at 0.6B<\/strong>, 1.7B<\/strong>, 4B<\/strong>, while the direct 1.58 bit baseline lags more as model size grows. On CPU, tokens per second<\/strong> improve by about 2.65\u00d7<\/strong>, and memory drops by about 10\u00d7<\/strong> for the student. The research team quantizes activations to INT8<\/strong> and uses the Straight Through Estimator<\/strong> for gradients through the quantizer. <\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.13998<\/figcaption><\/figure>\n<\/div>\n

The framework is compatible with post training quantization methods such as GPTQ<\/strong> and AWQ<\/strong>, which provide additional gains on top of the pipeline. Distilling from a stronger teacher helps more, which suggests pairing small 1.58 bit students with larger FP16 teachers when available.<\/p>\n

Key Takeaways<\/strong><\/h3>\n