{"id":45449,"date":"2025-10-19T07:10:03","date_gmt":"2025-10-19T07:10:03","guid":{"rendered":"https:\/\/youzum.net\/autocode-a-new-ai-framework-that-lets-llms-create-and-verify-competitive-programming-problems-mirroring-the-workflow-of-human-problem-setters\/"},"modified":"2025-10-19T07:10:03","modified_gmt":"2025-10-19T07:10:03","slug":"autocode-a-new-ai-framework-that-lets-llms-create-and-verify-competitive-programming-problems-mirroring-the-workflow-of-human-problem-setters","status":"publish","type":"post","link":"https:\/\/youzum.net\/fr\/autocode-a-new-ai-framework-that-lets-llms-create-and-verify-competitive-programming-problems-mirroring-the-workflow-of-human-problem-setters\/","title":{"rendered":"AutoCode: A New AI Framework that Lets LLMs Create and Verify Competitive Programming Problems, Mirroring the Workflow of Human Problem Setters"},"content":{"rendered":"

Are your LLM code benchmarks actually rejecting wrong-complexity solutions and interactive-protocol violations, or are they passing under-specified unit tests?<\/strong> A team of researchers from UCSD, NYU, University of Washington, Princeton University, Canyon Crest Academy, OpenAI, UC Berkeley, MIT, University of Waterloo, and Sentient Labs introduce AutoCode<\/strong>, a new AI framework that lets LLMs create and verify competitive programming problems, mirroring the workflow of human problem setters. AutoCode reframes evaluation for code-reasoning models by treating problem setting<\/strong> (not only problem solving) as the target task. The system trains LLMs to produce competition-grade statements<\/strong>, test data<\/strong>, and verdict logic<\/strong> that match official online judges at high rates. On a 7,538-problem benchmark built from prior datasets, AutoCode achieves 91.1% consistency<\/strong> with official judgments (FPR 3.7%, FNR 14.1%<\/strong>). On a separate, more difficult 720 recent Codeforces problems<\/strong> (including interactive tasks), the full framework reports 98.7% consistency<\/strong>, 1.3% FPR<\/strong>, 1.2% FNR<\/strong>. <\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.12803<\/figcaption><\/figure>\n<\/div>\n

Why problem setting matters for evaluation?<\/strong><\/h3>\n

Public code benchmarks often rely on under-specified tests that let wrong-complexity or shortcut solutions pass. That inflates scores and pollutes reinforcement signals (rewarding fragile tactics). AutoCode\u2019s validator-first<\/strong> approach and adversarial test generation aim to reduce false positives (FPR)<\/strong>\u2014incorrect programs that pass\u2014and false negatives (FNR)<\/strong>\u2014correct programs rejected due to malformed inputs. <\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.12803<\/figcaption><\/figure>\n<\/div>\n

The core loop: Validator \u2192 Generator \u2192 Checker<\/strong><\/h3>\n

AutoCode runs a closed loop that mirrors human contest workflows, but each step is selected from LLM-generated candidates<\/strong> using targeted in-framework tests.<\/p>\n

1) Validator (minimize FNR by enforcing input legality)<\/strong><\/p>\n

The system first asks an LLM to synthesize 40 evaluation inputs<\/strong>\u201410 valid<\/strong> and 30 near-valid illegal<\/strong> (e.g., off-by-one boundary violations). It then prompts the LLM for three candidate validator programs<\/strong> and selects the one that best classifies these cases. This prevents \u201ccorrect\u201d solutions from crashing on malformed data.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.12803<\/figcaption><\/figure>\n<\/div>\n

2) Generator (reduce FPR by adversarial coverage)<\/strong><\/p>\n

Three complementary strategies produce test cases:<\/strong>
\u2022 Small-data exhaustion<\/strong> for boundary coverage,
\u2022 Randomized + extreme<\/strong> cases (overflows, precision, hash-collisions),
\u2022 TLE-inducing<\/strong> structures to break wrong-complexity solutions.<\/p>\n

Invalid cases are filtered by the selected validator; then cases are deduplicated and bucket-balanced before sampling.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.12803<\/figcaption><\/figure>\n<\/div>\n

3) Checker (verdict logic)<\/strong><\/p>\n

The checker compares contestant outputs with the reference solution<\/strong> under complex rules. AutoCode again generates 40 checker scenarios<\/strong> and three candidate checker programs<\/strong>, keeps only scenarios with validator-approved inputs, and selects the best checker by accuracy against the 40 labeled scenarios.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.12803<\/figcaption><\/figure>\n<\/div>\n

4) Interactor (for interactive problems)<\/strong><\/p>\n

For tasks that require dialogue with the judge, AutoCode introduces a mutant-based interactor<\/strong>: it makes small logical edits (\u201cmutants\u201d) to the reference solution, selects interactors that accept the true solution<\/strong> but reject the mutants<\/strong>, maximizing discrimination. This addresses a gap in earlier public datasets that avoided interactives.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.12803<\/figcaption><\/figure>\n<\/div>\n

Dual verification enables new problems (not just tests for existing ones)<\/strong><\/h3>\n

AutoCode can generate novel problem variants<\/strong> starting from a random \u201cseed\u201d Codeforces problem (<2200 Elo). The LLM drafts a new statement and two solutions<\/strong>: an efficient reference<\/strong> and a simpler brute-force<\/strong> baseline. A problem is accepted only if the reference output matches<\/strong> brute force across the generated test suite (the brute force may TLE on large cases but serves as ground truth on small\/exhaustive cases). This dual-verification<\/strong> protocol filters ~27%<\/strong> of error-prone items, lifting reference-solution correctness from 86% \u2192 94%<\/strong> before human review.<\/p>\n

Human experts then grade the survivors on solvability, solution correctness, quality, novelty, difficulty<\/strong>. After filtering, 61.6%<\/strong> are usable for model training<\/strong>, 76.3%<\/strong> for human training<\/strong>, and 3.2%<\/strong> are ICPC\/IOI-level<\/strong> problems. Difficulty typically increases relative to the seed, and difficulty gain correlates with perceived quality.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.12803<\/figcaption><\/figure>\n<\/div>\n

Understanding the results<\/strong><\/h3>\n

Existing problems (7,538 total; 195,988 human submissions).<\/strong> AutoCode: 91.1% consistency<\/strong>, 3.7% FPR<\/strong>, 14.1% FNR<\/strong>, vs 72.9\u201381.0% consistency for prior generators (CodeContests, CodeContests+, TACO, HardTests).<\/p>\n

Recent Codeforces problems (720, unfiltered; includes interactives).<\/strong> AutoCode: 98.7% consistency<\/strong>, 1.3% FPR<\/strong>, 1.2% FNR<\/strong>. Ablations show all three generator strategies and prompt optimization<\/strong> contribute: removing prompt optimization drops consistency to 98.0%<\/strong> and more than doubles FNR<\/strong> to 2.9%<\/strong>.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.12803<\/figcaption><\/figure>\n<\/div>\n

Key Takeaways<\/strong><\/h3>\n