{"id":91207,"date":"2026-05-18T16:41:54","date_gmt":"2026-05-18T16:41:54","guid":{"rendered":"https:\/\/youzum.net\/nvidia-introduces-a-4-bit-pretraining-methodology-using-nvfp4-validated-on-a-12b-hybrid-mamba-transformer-at-10t-token-horizon\/"},"modified":"2026-05-18T16:41:54","modified_gmt":"2026-05-18T16:41:54","slug":"nvidia-introduces-a-4-bit-pretraining-methodology-using-nvfp4-validated-on-a-12b-hybrid-mamba-transformer-at-10t-token-horizon","status":"publish","type":"post","link":"https:\/\/youzum.net\/ja\/nvidia-introduces-a-4-bit-pretraining-methodology-using-nvfp4-validated-on-a-12b-hybrid-mamba-transformer-at-10t-token-horizon\/","title":{"rendered":"NVIDIA Introduces a 4-Bit Pretraining Methodology Using NVFP4, Validated on a 12B Hybrid Mamba-Transformer at 10T Token Horizon"},"content":{"rendered":"

Pretraining frontier-scale LLMs in FP8 is now standard practice, but moving to 4-bit floating point has remained an open research problem because narrower formats compress dynamic range and amplify quantization error at long token horizons. A new research from NVIDIA describes a pretraining methodology built around NVFP4<\/strong>, a 4-bit microscaling format supported natively by Blackwell Tensor Cores, and validates it by pretraining a 12-billion-parameter hybrid Mamba-Transformer on 10 trillion tokens<\/strong>. The research team state this is the longest publicly documented training run in 4-bit precision to date. The resulting model attains 62.58% on MMLU-Pro 5-shot versus 62.62% for the FP8 baseline, and is supported in NVIDIA\u2019s Transformer Engine.<\/p>\n

What NVFP4 Actually is<\/strong><\/h2>\n

To understand why NVFP4 is important, it helps to revisit how microscaling formats work. In a microscaling (MX) format, a contiguous block of low-precision elements shares a single scale factor, which is used to map the block back into a wider numerical range during the matrix multiply. MXFP4 uses 32-element blocks where each element is stored as E2M1 \u2014 1 sign bit, 2 exponent bits, 1 mantissa bit \u2014 encoding only the values \u00b10, \u00b10.5, \u00b11, \u00b11.5, \u00b12, \u00b13, \u00b14, and \u00b16. Block scale factors are stored in UE8M0, which restricts them to powers of two.<\/p>\n

NVFP4 changes three things. First, the block size drops from 32 to 16 elements, narrowing the dynamic range each scale has to cover. Second, block scale factors are stored in E4M3 rather than UE8M0, trading exponent range for mantissa precision so the per-block amax (absolute maximum) can be mapped much closer to the FP4 maximum representable. Third, NVFP4 adds a second scaling level: an FP32 per-tensor scale that remaps values so the E4M3 block scales themselves stay in range. The result is that at least 6.25% of values in each block \u2014 the per-block amax \u2014 are represented at near-FP8 precision, while the remainder sit in FP4.<\/p>\n

On NVIDIA Blackwell, FP4 GEMMs run at 4\u00d7 BF16 throughput on GB200 and 6\u00d7 on GB300, which translates to roughly 2\u00d7 and 3\u00d7 speedups over FP8. Operand memory footprint is approximately halved compared to FP8.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2509.25149<\/figcaption><\/figure>\n<\/div>\n

What\u2019s Quantized \u2014 and What Isn\u2019t<\/strong><\/h2>\n

Only the GEMMs inside linear (fully-connected) layers Fprop, Dgrad, and Wgrad actually run in NVFP4. Embeddings, the output projection head, normalization layers, non-linearities, and all attention components (softmax and the query-key and attention score-value batched GEMMs) stay in BF16 or FP32. Model weights, weight gradients used for accumulation across microbatches and data-parallel replicas, and optimizer states are kept in FP32. Tensor parallel reductions run in BF16. <\/p>\n

The Four-Part Training Methodology<\/strong><\/h2>\n

Quantizing every linear-layer GEMM to NVFP4 with default settings (1\u00d716 block scaling everywhere, round-to-nearest-even on every tensor, no transforms) diverges early in training. NVIDIA\u2019s approach stabilizes it with four components, and ablation studies on the 12B model show each is necessary.<\/p>\n

Selective high precision:<\/strong> Linear layers in the first two and the final eight of the 62 blocks (about 16% of all linear layers) are kept in BF16. Ablations indicated that the final blocks are the sensitive ones because they require more dynamic range than FP4 provides; keeping only the final four blocks in BF16 was also enough for stable convergence.<\/p>\n

Random Hadamard Transforms (RHT):<\/strong> Outliers in weight gradients are spread into an approximately Gaussian distribution by multiplying the input tiles with a 16\u00d716 Hadamard matrix combined with a random \u00b11 sign vector. Because the orthogonal transforms cancel inside the dot-product, no math correction is needed in the GEMM. The d=16 size was chosen empirically: d=4 hurt convergence, d=128 gave similar results. RHT is applied only to the inputs of the weight-gradient (Wgrad) GEMM, and a single random sign vector is shared across all linear layers. Randomization itself was a no-op at the 1.2B scale but measurably improved the 12B run.<\/p>\n

Two-dimensional (2D) block scaling for weights<\/strong>: Standard NVFP4 scales 1\u00d716 blocks along the dot-product dimension. Because the backward pass transposes the weight tensor, the forward and backward passes end up with different quantized weights, breaking the chain rule. NVIDIA\u2019s fix is to scale weights in 16\u00d716 blocks so the same quantized representation is used in both passes. Activations and gradients keep 1\u00d716 scaling, since they are less sensitive to this inconsistency.<\/p>\n

Stochastic rounding on gradients<\/strong>: Round-to-nearest-even introduces systematic bias when applied to gradient tensors. Stochastic rounding rounds probabilistically based on distance to the two nearest representable values, removing that bias. The research team explicitly notes in research paper<\/a> that stochastic rounding is detrimental<\/strong> when applied to forward-pass tensors, so it is restricted to gradients.<\/p>\n

Results on the 12B Hybrid Mamba-Transformer<\/strong><\/h2>\n

The 12B model uses the Nemotron-Nano-12B-v2-Base architecture \u2014 62 blocks (6 Self-Attention, 28 FFN, 28 Mamba-2), hidden dimension 5120, FFN dimension 20480 \u2014 trained with a Warmup-Stable-Decay schedule (constant LR through 80% of training, decay over the final 20%), batch size 736, sequence length 8192. The FP8 reference baseline follows the DeepSeek-V3 methodology: E4M3 elements, 128\u00d7128 weight blocks, 1\u00d7128 activation and gradient blocks, with the first block and last two blocks kept in BF16.<\/p>\n

NVFP4 validation loss stays within 1% of the FP8 baseline during the stable phase and widens to slightly above 1.5% during decay. Downstream accuracy is comparable across most benchmarks: MMLU 76.57% vs 77.36%, GSM8K CoT 92.27% vs 89.08%, MATH 81.48% vs 83.32%, AGIEval English CoT 70.31% vs 67.01%. Coding shows the largest gap \u2014 HumanEval+ 57.43% vs 59.93%, MBPP+ 55.91% vs 59.11% \u2014 which the research team attributes partly to noisy final-checkpoint evaluation. The research team also documents a precision-switching technique: transitioning the forward pass from NVFP4 to BF16 starting at 8.2T tokens (about 18% of the schedule) reduced relative loss error from 1.5% to 0.5%.<\/p>\n

NVFP4 vs MXFP4<\/strong><\/h2>\n

On a separate 8B hybrid Mamba-Transformer trained on 1T tokens, NVFP4 reached a relative loss error of about 1.5% versus BF16, while MXFP4 stayed near 2.5%. To close the gap, MXFP4 required 1.36T tokens to match the NVFP4 1T-token loss \u2014 a 36% token overhead. The research team attributes the difference to NVFP4\u2019s smaller block size and E4M3 scales, which preserve more of the FP4 dynamic range than MXFP4\u2019s power-of-two UE8M0 scales (which can waste up to one binade and the \u00b14, \u00b16 samples in the worst case).<\/p>\n

Marktechpost\u2019s Visual Explainer<\/strong><\/h2>\n
\n

<\/p>\n

\n
\n
<\/span><\/span><\/span><\/div>\n

NVFP4 Pretraining Guide<\/span>\n <\/p><\/div>\n

01 \/ 10<\/span>\n <\/p><\/div>\n

<\/p>\n

\n
\n

<\/p>\n

\n
\n \u25cf NVIDIA Technical Report<\/span>\n

Pretraining Large Language Models with NVFP4<\/h1>\n

A 4-bit floating-point training recipe validated on a 12-billion-parameter hybrid Mamba-Transformer trained on 10 trillion tokens \u2014 the longest publicly documented 4-bit pretraining run to date.<\/p>\n

\n
\n
12B<\/div>\n
Parameters<\/div>\n<\/div>\n
\n
10T<\/div>\n
Training Tokens<\/div>\n<\/div>\n
\n
62.58%<\/div>\n
MMLU-Pro (vs 62.62 FP8)<\/div>\n<\/div>\n<\/div>\n
SOURCE \u2014 arXiv:2509.25149v2<\/span> \u00b7 NVIDIA \u00b7 Available in Transformer Engine<\/div>\n<\/div>\n<\/section>\n

<\/p>\n

\n
01 \u2014 Context<\/div>\n

Why move from FP8 to 4-bit pretraining<\/h2>\n

FP8 training is now standard for frontier LLM pretraining. Moving to FP4 promises a 2\u00d7 to 3\u00d7 boost in arithmetic throughput<\/strong> over FP8 and approximately half the operand memory \u2014 but narrower formats compress dynamic range and amplify quantization error at long token horizons.<\/p>\n

The challenge is to preserve training stability and downstream accuracy across multi-trillion-token runs. This report presents a recipe that does both, using NVFP4<\/strong>, a 4-bit microscaling format with native support on NVIDIA Blackwell Tensor Cores.<\/p>\n

\n
\n
GB200 Throughput<\/div>\n