{"id":44765,"date":"2025-10-16T07:02:47","date_gmt":"2025-10-16T07:02:47","guid":{"rendered":"https:\/\/youzum.net\/qerl-nvfp4-quantized-reinforcement-learning-rl-brings-32b-llm-training-to-a-single-h100-while-improving-exploration\/"},"modified":"2025-10-16T07:02:47","modified_gmt":"2025-10-16T07:02:47","slug":"qerl-nvfp4-quantized-reinforcement-learning-rl-brings-32b-llm-training-to-a-single-h100-while-improving-exploration","status":"publish","type":"post","link":"https:\/\/youzum.net\/es\/qerl-nvfp4-quantized-reinforcement-learning-rl-brings-32b-llm-training-to-a-single-h100-while-improving-exploration\/","title":{"rendered":"QeRL: NVFP4-Quantized Reinforcement Learning (RL) Brings 32B LLM Training to a Single H100\u2014While Improving Exploration"},"content":{"rendered":"

What would you build if you could run <\/strong>Reinforcement Learning<\/strong> (RL) post-training on a 32B LLM in 4-bit NVFP4\u2014on a single H100\u2014with BF16-level accuracy and 1.2\u20131.5\u00d7 step speedups? <\/strong>NVIDIA researchers (with collaborators from MIT, HKU, and Tsinghua) have open-sourced QeRL (Quantization-enhanced Reinforcement Learning)<\/strong>, a training framework that pushes Reinforcement Learning<\/strong> (RL) post-training into 4-bit FP4 (NVFP4)<\/strong> while keeping gradient math in higher precision via LoRA. The research team reports >1.5\u00d7 speedups in the rollout phase<\/strong>, ~1.8\u00d7 end-to-end vs QLoRA<\/strong> in one setting, and the first demonstration of RL training for a 32B policy on a single H100-80GB GPU<\/strong>. <\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.11696<\/figcaption><\/figure>\n<\/div>\n

What QeRL changes in the Reinforcement Learning (RL) loop<\/strong>?<\/h3>\n

Most RLHF\/GRPO\/DAPO pipelines spend the bulk of wall-clock time in rollouts<\/strong> (token generation). QeRL shifts the policy\u2019s weight path to NVFP4 (FP4) with dual-level scaling<\/strong> and keeps logits\/gradients in higher precision via LoRA<\/strong>, so backprop remains stable while the sampling path hits hardware-efficient FP4\u00d7BF16 kernels (Marlin). The result is faster prefill\/decoding during rollouts without maintaining a separate full-precision policy. <\/p>\n

Mechanically, the research team integrates Marlin-based FP4 kernels<\/strong> in both rollout and prefill, while LoRA limits trainable parameters. This directly targets the stage that dominates RL cost and latency for long reasoning traces.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.11696<\/figcaption><\/figure>\n<\/div>\n

Quantization as exploration, made schedulable<\/strong><\/h3>\n

A core empirical finding: deterministic FP4 quantization raises policy entropy<\/strong>, flattening token distributions early in training and improving exploration<\/strong> versus 16-bit LoRA and NF4-based QLoRA baselines. To control that effect over time, QeRL introduces Adaptive Quantization Noise (AQN)<\/strong>\u2014channel-wise Gaussian perturbations<\/strong> mapped into LayerNorm scale parameters<\/strong> and annealed with an exponential schedule<\/strong>. This keeps kernel fusion intact (no extra weight tensors) while transitioning from exploration to exploitation.<\/p>\n

\n
\"\"<\/figure>\n<\/div>\n

In ablations, QeRL shows faster reward growth<\/strong> and higher final scores<\/strong> on math-reasoning tasks under both GRPO<\/strong> and DAPO<\/strong>, aligning with the hypothesis that structured noise in parameter space can be a useful exploration driver in RL, even though such noise is typically detrimental in supervised fine-tuning.<\/p>\n

Reported results<\/strong><\/h3>\n

On Qwen2.5<\/strong> backbone model, the research team show that NVFP4+LoRA<\/strong> outperforms vanilla LoRA<\/strong> and QLoRA<\/strong> in rollout throughput and overall training time, with >2\u00d7 rollout throughput on 14B\/32B<\/strong> models against QLoRA and ~1.8\u00d7 end-to-end vs QLoRA<\/strong> in a representative setup. They also demonstrate training a 32B policy with GRPO on a single H100-80GB<\/strong>, enabled by the lower memory footprint of weight-only FP4.<\/p>\n

Accuracy is competitive with higher-precision baselines. For a 7B<\/strong> model, the research team reports GSM8K = 90.8%<\/strong> and MATH500 = 77.4%<\/strong>, surpassing 16-bit LoRA and QLoRA<\/strong> under their setup and matching full-parameter fine-tuning<\/strong>. Across broader math benchmarks (e.g., BigMath), QeRL maintains parity or advantage, while converging faster due to improved exploration. <\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.11696<\/figcaption><\/figure>\n<\/div>\n

What this is\u2014and isn\u2019t?<\/strong><\/h3>\n

QeRL is weight-only FP4<\/strong> with LoRA updates<\/strong>; it does not<\/strong> claim FP4 precision for logits\/gradients. The benefits concentrate in rollout\/prefill throughput<\/strong> and memory footprint<\/strong>, with empirical evidence that quantization-induced entropy<\/strong> aids RL exploration when AQN<\/strong> modulates it over training. Generalization to modalities beyond math-reasoning tasks or to safety\/tool-use RL depends on reward design and sequence lengths.<\/p>\n

Key Takeaways<\/strong><\/h3>\n