AI

LLMs have made impressive strides in generating code for various programming tasks. However, they mostly rely on recognizing patterns from static code examples rather than understanding how the code behaves during execution. This often leads to programs that look correct but fail when run. While recent methods introduce iterative refinement and self-debugging, they typically act in separate steps, generating, testing, and then revising. Unlike human programmers who constantly run fragments of code and adjust based on real-time output, these models cannot integrate execution feedback continuously, limiting their effectiveness in producing truly functional code. The Role of Program Synthesis and Prompting in Code Generation Program synthesis has long been used to evaluate LLMs and automate code generation benchmarks, such as MBPP, HumanEval, and CodeContests, by testing models on various coding challenges. While prompting strategies, such as few-shot and Chain-of-Thought, have improved performance, newer methods now incorporate feedback loops that utilize tools or execution results to refine outputs. Some frameworks even assign tasks to multiple LLM agents, each tackling different aspects of the problem. However, most approaches still rely on simple decoding methods. Unlike traditional strategies, newer guidance techniques, such as CFG, offer a more dynamic approach but haven’t yet been widely applied with real-time execution feedback. Introducing EG-CFG: Execution-Guided Code Generation from Tel Aviv University Researchers at Tel Aviv University have introduced EG-CFG, a new method for code generation that actively utilizes execution feedback during the generation process, a technique commonly employed by human programmers. Instead of waiting until the end, EG-CFG evaluates partial code as it’s being written, guiding the model toward correct and executable outputs. It uses a beam search to generate multiple code options, runs them, and integrates runtime outcomes to influence the next steps. This real-time feedback loop significantly boosts performance across standard benchmarks, such as MBPP, HumanEval, and CodeContests, even surpassing closed-source models, while also enabling efficient parallel reasoning and dynamic exploration. How EG-CFG Works: Real-Time Feedback Meets Beam Search and AST Parsing The EG-CFG method improves code generation by guiding language models using real-time execution feedback during inference. For a given programming task, it generates partial code solutions and explores multiple continuations using beam search. These continuations are checked for syntax using AST parsing, and only valid ones are executed on test cases to gather detailed runtime traces, including variable states and errors. This feedback is then injected into the model’s prompt to inform future predictions. A guidance mechanism interpolates between the model’s standard output and feedback-informed suggestions, helping the model refine its solution step by step until it passes all test cases. Benchmark Results: EG-CFG Outperforms GPT-4 and Claude on HumanEval and MBPP-ET The EG-CFG method was tested using two versions of DeepSeek models: a 1.3B parameter model locally and the larger V3-0324 model through an API. It was evaluated on five code benchmarks: MBPP, HumanEval, CodeContests, MBPP-ET, and HumanEval-ET. On HumanEval, EG-CFG with DeepSeek V3 solved 90.1% of the tasks correctly, outperforming GPT-4 (85.5%) and Claude 2 (83.2%). On MBPP-ET, it achieved an 81.4% accuracy rate, setting a new benchmark. Notably, the smaller 1.3B model also showed strong gains, improving from 46.3% to 61.7% on HumanEval when guided with EG-CFG. An ablation study confirmed the importance of components like dynamic feedback and beam search in driving these results. Conclusion: EG-CFG Simulates Human Debugging to Advance Code Generation In conclusion, the EG-CFG method introduces a new way to generate code using language models by incorporating real-time execution feedback during generation. Unlike traditional approaches that rely on static patterns, EG-CFG simulates how human programmers test and refine code. It uses beam search to explore possible code completions, tests them with real inputs, and then guides generation based on the results. This happens line by line, ensuring feedback is both structured and actionable. The method also supports multiple agents working in parallel, boosting efficiency. EG-CFG achieves top accuracy across standard benchmarks, showing strong results even on complex coding tasks and with smaller models. Check out the Paper and GitHub Page. All credit for this research goes to the researchers of this project. Sponsorship Opportunity Reach the most influential AI developers worldwide. 1M+ monthly readers, 500K+ community builders, infinite possibilities. [Explore Sponsorship] The post EG-CFG: Enhancing Code Generation with Real-Time Execution Feedback appeared first on MarkTechPost.

The development of large-scale language models (LLMs) has historically required centralized access to extensive datasets, many of which are sensitive, copyrighted, or governed by usage restrictions. This constraint severely limits the participation of data-rich organizations operating in regulated or proprietary environments. FlexOlmo—introduced by researchers at the Allen Institute for AI and collaborators—proposes a modular training and inference framework that enables LLM development under data governance constraints. Current LLMs….. Current LLM training pipelines rely on aggregating all training data into a single corpus, which imposes a static inclusion decision and eliminates the possibility of opt-out post-training. This approach is incompatible with: Regulatory regimes (e.g., HIPAA, GDPR, data sovereignty laws), License-bound datasets (e.g., non-commercial or attribution-restricted), Context-sensitive data (e.g., internal source code, clinical records). FlexOlmo addresses two objectives: Decentralized, modular training: Allow independently trained modules on disjoint, locally held datasets. Inference-time flexibility: Enable deterministic opt-in/opt-out mechanisms for dataset contributions without retraining. Model Architecture: Expert Modularity via Mixture-of-Experts (MoE) FlexOlmo builds upon a Mixture-of-Experts (MoE) architecture where each expert corresponds to a feedforward network (FFN) module trained independently. A fixed public model (denoted as M_pub) serves as the shared anchor. Each data owner trains an expert M_i using their private dataset D_i, while all attention layers and other non-expert parameters remain frozen. Key architectural components: Sparse activation: Only a subset of expert modules is activated per input token. Expert routing: Token-to-expert assignment is governed by a router matrix derived from domain-informed embeddings, eliminating the need for joint training. Bias regularization: A negative bias term is introduced to calibrate selection across independently trained experts, preventing over-selection of any single expert. This design maintains interoperability among modules while enabling selective inclusion during inference. Asynchronous and Isolated Optimization Each expert M_i is trained via a constrained procedure to ensure alignment with M_pub. Specifically: Training is performed on a hybrid MoE instance comprising M_i and M_pub. The M_pub expert and shared attention layers are frozen. Only the FFNs corresponding to M_i and the router embeddings r_i are updated. To initialize r_i, a set of samples from D_i is embedded using a pretrained encoder, and their average forms the router embedding. Optional lightweight router tuning can further improve performance using proxy data from the public corpus. Dataset Construction: FLEXMIX The training corpus, FLEXMIX, is divided into: A public mix, composed of general-purpose web data. Seven closed sets simulating non-shareable domains: News, Reddit, Code, Academic Text, Educational Text, Creative Writing, and Math. Each expert is trained on a disjoint subset, with no joint data access. This setup approximates real-world usage where organizations cannot pool data due to legal, ethical, or operational constraints. Evaluation and Baseline Comparisons FlexOlmo was evaluated on 31 benchmark tasks across 10 categories, including general language understanding (e.g., MMLU, AGIEval), generative QA (e.g., GEN5), code generation (e.g., Code4), and mathematical reasoning (e.g., Math2). Baseline methods include: Model soup: Averaging weights of individually fine-tuned models. Branch-Train-Merge (BTM): Weighted ensembling of output probabilities. BTX: Converting independently trained dense models into a MoE via parameter transplant. Prompt-based routing: Using instruction-tuned classifiers to route queries to experts. Compared to these methods, FlexOlmo achieves: A 41% average relative improvement over the base public model. A 10.1% improvement over the strongest merging baseline (BTM). The gains are especially notable on tasks aligned with closed domains, confirming the utility of specialized experts. Architectural Analysis Several controlled experiments reveal the contribution of architectural decisions: Removing expert-public coordination during training significantly degrades performance. Randomly initialized router embeddings reduce inter-expert separability. Disabling the bias term skews expert selection, particularly when merging more than two experts. Token-level routing patterns show expert specialization at specific layers. For instance, mathematical input activates the math expert at deeper layers, while introductory tokens rely on the public model. This behavior underlines the model’s expressivity compared to single-expert routing strategies. Opt-Out and Data Governance A key feature of FlexOlmo is deterministic opt-out capability. Removing an expert from the router matrix fully removes its influence at inference time. Experiments show that removing the News expert reduces performance on NewsG but leaves other tasks unaffected, confirming the localized influence of each expert. Privacy Considerations Training data extraction risks were evaluated using known attack methods. Results indicate: 0.1% extraction rate for a public-only model. 1.6% for a dense model trained on the math dataset. 0.7% for FlexOlmo with the math expert included. While these rates are low, differential privacy (DP) training can be applied independently to each expert for stronger guarantees. The architecture does not preclude the use of DP or encrypted training methods. Scalability The FlexOlmo methodology was applied to an existing strong baseline (OLMo-2 7B), pretrained on 4T tokens. Incorporating two additional experts (Math, Code) improved average benchmark performance from 49.8 to 52.8, without retraining the core model. This demonstrates scalability and compatibility with existing training pipelines. Conclusion FlexOlmo introduces a principled framework for building modular LLMs under data governance constraints. Its design supports distributed training on locally maintained datasets and enables inference-time inclusion/exclusion of dataset influence. Empirical results confirm its competitiveness against both monolithic and ensemble-based baselines. The architecture is particularly applicable to environments with: Data locality requirements, Dynamic data use policies, Regulatory compliance constraints. FlexOlmo provides a viable pathway for constructing performant language models while adhering to real-world data access boundaries. Check out the Paper, Model on Hugging Face and Codes. All credit for this research goes to the researchers of this project. Sponsorship Opportunity: Reach the most influential AI developers in US and Europe. 1M+ monthly readers, 500K+ community builders, infinite possibilities. [Explore Sponsorship] The post You Don’t Need to Share Data to Train a Language Model Anymore—FlexOlmo Demonstrates How appeared first on MarkTechPost.

Google’s new Gemini Embedding model now leads the MTEB benchmark. But it is facing fierce competition from closed and open source rivals.Read More