{"id":43177,"date":"2025-10-09T06:56:26","date_gmt":"2025-10-09T06:56:26","guid":{"rendered":"https:\/\/youzum.net\/ra3-mid-training-with-temporal-action-abstractions-for-faster-reinforcement-learning-rl-post-training-in-code-llms\/"},"modified":"2025-10-09T06:56:26","modified_gmt":"2025-10-09T06:56:26","slug":"ra3-mid-training-with-temporal-action-abstractions-for-faster-reinforcement-learning-rl-post-training-in-code-llms","status":"publish","type":"post","link":"https:\/\/youzum.net\/th\/ra3-mid-training-with-temporal-action-abstractions-for-faster-reinforcement-learning-rl-post-training-in-code-llms\/","title":{"rendered":"RA3: Mid-Training with Temporal Action Abstractions for Faster Reinforcement Learning (RL) Post-Training in Code LLMs"},"content":{"rendered":"

TL;DR:<\/strong> A new research from Apple, formalizes what \u201cmid-training\u201d should do before reinforcement learning RL post-training and introduces RA3 (Reasoning as Action Abstractions)<\/strong>\u2014an EM-style procedure that learns temporally consistent latent actions from expert traces, then fine-tunes on those bootstrapped traces. It shows mid-training should (1) prune to a compact near-optimal action subspace and (2) shorten the effective planning horizon, improving RL convergence. Empirically, RA3 improves HumanEval\/MBPP by ~8\/4 points over base\/NTP and accelerates RLVR on HumanEval+, MBPP+, LiveCodeBench, and Codeforces. <\/p>\n

What does the research present?<\/strong><\/h3>\n

The research team present the first formal treatment of how mid-training shapes post-training reinforcement learning RL: they breakdown outcomes into (i) pruning efficiency<\/strong>\u2014how well mid-training selects a compact near-optimal action subset that shapes the initial policy prior\u2014and (ii) RL convergence<\/strong>\u2014how quickly post-training improves within that restricted set. The analysis argues mid-training is most effective when the decision space is compact<\/strong> and the effective horizon is short<\/strong>, favoring temporal abstractions<\/strong> over primitive next-token actions.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2509.25810<\/figcaption><\/figure>\n<\/div>\n

Algorithm: RA3 in one pass<\/strong><\/h3>\n

RA3<\/strong> derives a sequential variational lower bound<\/strong> (a temporal ELBO) and optimizes it with an EM-like loop:<\/strong><\/p>\n