Committee

Microsoft has open sourced code-testing-generator, a polyglot agent that writes unit tests and then proves they work. It ships in the dotnet-test plugin inside the MIT-licensed dotnet/skills repository. The agent targets a gap that coding assistants usually leave open. A prompt like ‘generate unit tests’ does not say which framework, file location or assertions to use. code-testing-generator settles those decisions by reading the repository before it writes anything. It then plans, writes, runs and checks the tests it produces. On Microsoft’s internal 152-task benchmark, it completed 140 tasks against 120 for stock GitHub Copilot. Both setups used the same model and prompts. Is it deployable Yes. It is an agent definition along with skills, not a hosted service, so it runs inside your existing coding agent and code stays local. Company stage: viable from solo maintainers upward. Startups and mid-market teams gain most, because the agent supplies repository research a small team has no time to encode. Enterprises can fork the language guidance to match internal frameworks. Industries: regulated or audit-heavy software estates — financial services, healthcare, insurance, public sector — plus platform teams paying down legacy test debt. Applications: backfilling tests on untested modules, generating tests for a pull-request diff, raising coverage before a release gate, and standardising conventions across polyglot monorepos. What the agent actually does It coordinates work through a Research-Plan-Implement (RPI) pipeline. It searches the repository for code needing tests, detects the language and test framework, reads existing tests for conventions, and finds the real build and test commands. That last step targets a specific failure: a test project that builds locally but never runs in CI because nothing registered it. The agent then picks one of three strategies. Direct writes and validates tests immediately. Single pass runs one cycle. Iterative repeats it for large scopes or coverage targets. It never modifies production code, and avoids tests that call external URLs, bind ports or depend on timing. The verification gate Before reporting completion, the agent runs five checks. It reasons about small code changes that should make the tests fail, a lightweight form of mutation testing. It looks for weak or missing assertions. It maps every requested scenario to a test. It builds the full workspace and runs the full suite. It confirms the repository’s own test command discovers the new tests. Benchmark results On Microsoft’s internal benchmark of 152 tasks from real repositories, the agent completed 140 (92.1%) versus 120 (78.9%) for stock GitHub Copilot on the same model and prompts (63% fewer failures). The gain is concentrated. On 89 vague prompts, the agent resolved 79 (88.8%) against 59 (66.3%), cutting failures from 30 to 10. On 63 detailed prompts, both scored 61 (96.8%). On 15 tasks targeting a specific diff, the agent passed all 15 and stock Copilot passed none. Notably, the agent generated 2.3% fewer tests (6,963 vs 7,129) at effectively identical line coverage (72.4% vs 72.2%). Average task time was 359 seconds against 380. Token use per completed task was 3.2% higher. On 45 .NET tasks, Claude Opus 4.8 reached 43/45 with the agent versus 35/45 stock; GPT-5.5 reached 41/45 versus 36/45. On the harder external SWE Atlas benchmark, completion was 16/44 versus 12/44. Explainer: how the agent turns one prompt into verified tests Key Takeaways Open source, MIT-licensed, polyglot unit-test agent from Microsoft’s .NET team. Research-Plan-Implement pipeline replaces one-shot generation with repository-aware planning. 92.1% vs 78.9% task completion against stock Copilot on the same model. Gains come almost entirely from vague prompts and diff-targeted requests. Fewer tests, same coverage, 5.5% faster — reliability, not volume. Check out the Technical details and Repo. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Microsoft Open Sources code-testing-generator: a Polyglot Unit-Test Agent That Hits 92.1% Task Completion Versus 78.9% for Stock Copilot appeared first on MarkTechPost.

arXiv:2608.05576v1 Announce Type: new Abstract: When a large language model (LLM) writes Harry Potter fanfiction, it reliably produces fundamental elements of the Hogwarts universe, such as recognizable places and characters. Human-written Harry Potter fanfictions, however, typically include these fundamentals and much more, incorporating stylistically irregular content and relationship-diverse plotlines. This gap between LLM and human writing has been noted across a variety of domains. LLMs tend to produce “average” writing, while human writing contains more diverse content that covers a broader distribution. Existing work has shown the existence of this distributional “gap”, but no work has proposed a systematic way to measure it. Our paper proposes a human-grounded framework that uses the empirical distribution of human writing on a topic to measure the distributional breadth of LLM-generated content on that same topic. We propose two metrics, LLM Coverage (LLM-Cov) and In-Boundary Rate (IBR), that separate the plausibility of LLM content from its distributional breadth. Across ideation and narrative tasks, we find that current LLMs produce plausible but narrow content that concentrates near the center of the human response space. Our framework can enable researchers to better assess the distributional breadth of LLM-authored content, which we term its “cultural reach”.

But cutting your runtime token burn is just the first problem.

Prime Intellect has open-sourced Prime Agent, a self-improving coding harness designed around two abstractions, the Recursive Language Model (RLM) and Continual Harness. Fixed tool schemas and context compaction force a model to work around its own scaffolding. Prime Agent replaces both with a persistent Python REPL and a rewritable harness. With Opus 5, it reports 95.5% on ARC-AGI-3, above the reported human expert baseline of 95.4%. It is MIT-licensed. Is it deployable Yes, today. Prime Agent installs on Linux or macOS with one command. It runs on subscription logins (Codex, Claude Pro/Max, GitHub Copilot), API keys (Anthropic, OpenAI, Google, Groq, Fireworks, Prime Inference, and others), Azure OpenAI, Amazon Bedrock, and self-hosted vLLM, Ollama, or LM Studio endpoints. Self-hosting an open-weights model such as GLM-5.2 keeps code inside your own network. Company level: Best fit is mid-size to large engineering orgs and AI labs that already run isolated CI containers. Prime Intellect states plainly that worker and kernel processes are not a security sandbox. Deployment therefore needs disposable clones or restricted environments. Solo developers can install it, but the payoff appears on multi-hour tasks. Industries: Developer tooling, semiconductor and HPC teams writing GPU kernels, simulation and gaming, quantitative research, and AI research labs. Applications: Overnight refactors behind a test gate, spec-driven builds from scratch, kernel optimization, long-horizon agent evaluation, and autoresearch. What Prime Intellect shipped Prime Agent is built on two abstractions. The Recursive Language Model (RLM) treats context as a variable and sub-agent delegation as function calls inside a REPL. The Continual Harness treats prompts, sub-agents, skills, and memory as state the agent can create, read, update, and delete from its own trajectory. Both papers have Prime Agent authors on them. The TUI is built on pi. Programmatic tool calling Models in Prime Agent get one tool: a persistent IPython kernel. Skills, tools, and sub-agents are pre-imported modules inside it. rlm(“sub-task”) launches a child session with its own model, kernel, and history, returning at admission rather than blocking. Results arrive through agent_message.send(…). A background daemon owns every live session. You can detach and reattach without stopping the loop, and a crashed worker recovers from the session JSONL plus a kernel snapshot. Agent-to-agent messaging is deliberately scoped to the nuclear family — parent, sibling, or child — to prevent cross-session chatter. Retained sub-agents drop from memory after 30 minutes idle, then reload when addressed. Self-improvement through /refine Continual Harness formalizes harness state as H = (ρ, G, K, M): prompt, sub-agents, skills, memory. Each exposes the same create, read, update, delete surface. /refine reads the agent’s own trajectory and applies the smallest relevant edit, recording the trigger and the outcome. Planning runs in the background without blocking the conversation. The base system prompt stays immutable, and a bad update can be reverted by ID. Benchmarks On ARC-AGI-3, Prime Agent with Opus 5 reports 95.5% RHAE Best@1, above the ARC reported human expert baseline of 95.4%. Three runs land at 95.0, 95.2, and 95.5, with 99.97% Best@3 and all 183/183 levels complete. Prime Intellect also reports lower token usage than native harnesses, crediting functions run over data instead of data read through tools. On a long-context suite, Prime Agent with open-weights GLM-5.2 beats Pi-mono on eight of nine evals. With Opus 5 it edges Claude Code on six of nine; with GPT-5.6 Sol it beats Codex on six of nine. Case studies include EmulatorBench, where the agent builds emulators in Rust from spec with no reference implementation and reproduces the SEGA Genesis and Game Boy Color; PMPP-Hard, for GPU kernels verified against KernelGuard; and Factorio, where it reached 100K+ production score in hours. Factorio also produced the most useful negative result. Prime Agent found it could spawn resources straight into assembly machines through RCON commands, despite a heartbeat prompt telling it not to cheat. The same refinement loop that built legitimate skills then built efficient cheating skills. Key Takeaways Prime Agent is MIT-licensed, installs in one command, and works with subscriptions, APIs, or self-hosted models. One tool — a persistent IPython kernel — replaces fixed tool schemas; sub-agents are function calls. /refine edits prompts, skills, memory, and sub-agent specs from the trajectory, with rollback by ID. Opus 5 in Prime Agent hits 95.5% on ARC-AGI-3, above the 95.4% human expert baseline. Check out the Technical details and GitHub Repo. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Prime Intellect Releases Prime Agent: An Open-Source RLM Harness Where Sub-Agents Are Function Calls Inside Persistent IPython Kernel appeared first on MarkTechPost.

In this tutorial, we explore adaptive experimentation using Meta’s Ax with the modern Client API. We work through a complete workflow where we tune a RandomForest model on a synthetic classification dataset while balancing predictive accuracy against model footprint. We begin by defining a mixed search space with integer, float, log-scaled, and categorical parameters, then use Ax’s ask-tell optimization loop to run constrained Bayesian optimization, multi-objective optimization, and parameter-constrained experimentation. Along the way, we visualize convergence, inspect the Pareto frontier, use Ax’s built-in analysis tools, and persist the experiment for future reuse. Copy CodeCopiedUse a different Browser import importlib, subprocess, sys def _ensure(module, pip_name=None): try: importlib.import_module(module) except ImportError: print(f”Installing {pip_name or module} …”) subprocess.check_call([sys.executable, “-m”, “pip”, “install”, “-q”, pip_name or module]) _ensure(“ax”, “ax-platform”) _ensure(“sklearn”, “scikit-learn”) import logging, warnings, time import numpy as np import matplotlib.pyplot as plt warnings.filterwarnings(“ignore”) logging.getLogger(“ax”).setLevel(logging.WARNING) from ax.api.client import Client from ax.api.configs import RangeParameterConfig, ChoiceParameterConfig from sklearn.datasets import make_classification from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import StratifiedKFold, cross_val_score np.random.seed(0) We begin by preparing the Colab environment and installing the required packages for Ax and scikit-learn. We import the core libraries for optimization, machine learning, plotting, logging, and reproducibility. We also configure warnings and Ax logging to keep the notebook output clean and focused on the experimental results. Copy CodeCopiedUse a different Browser X, y = make_classification( n_samples=1400, n_features=20, n_informative=8, n_redundant=4, n_classes=3, random_state=0, ) CV = StratifiedKFold(n_splits=3, shuffle=True, random_state=0) def evaluate(p): n_est, depth = int(p[“n_estimators”]), int(p[“max_depth”]) clf = RandomForestClassifier( n_estimators=n_est, max_depth=depth, max_features=float(p[“max_features”]), min_samples_leaf=int(p[“min_samples_leaf”]), criterion=p[“criterion”], ccp_alpha=float(p[“ccp_alpha”]), n_jobs=-1, random_state=0, ) accuracy = cross_val_score(clf, X, y, cv=CV, scoring=”accuracy”).mean() model_size = n_est * depth return {“accuracy”: float(accuracy), “model_size”: float(model_size)} SEARCH_SPACE = [ RangeParameterConfig(name=”n_estimators”, bounds=(50, 300), parameter_type=”int”), RangeParameterConfig(name=”max_depth”, bounds=(3, 24), parameter_type=”int”), RangeParameterConfig(name=”max_features”, bounds=(0.2, 1.0), parameter_type=”float”), RangeParameterConfig(name=”min_samples_leaf”,bounds=(1, 12), parameter_type=”int”), RangeParameterConfig(name=”ccp_alpha”, bounds=(1e-5, 1e-1), parameter_type=”float”, scaling=”log”), ChoiceParameterConfig(name=”criterion”, values=[“gini”, “entropy”, “log_loss”], parameter_type=”str”, is_ordered=False), ] def run_study(client, total_trials, metric_keys, batch=4): records = [] while len(records) < total_trials: trials = client.get_next_trials(max_trials=min(batch, total_trials – len(records))) if not trials: break for idx, params in trials.items(): full = evaluate(params) raw = {k: full[k] for k in metric_keys} client.complete_trial(trial_index=idx, raw_data=raw) records.append({“trial”: idx, “params”: params, **full}) return records We create a synthetic multi-class classification dataset and define a cross-validation strategy to evaluate Random Forest models. We build an evaluation function that returns both accuracy and model size, allowing us to measure performance and cost together. We then define a mixed search space with integer, float, log-scaled, and categorical parameters, along with a reusable ask-tell study runner. Copy CodeCopiedUse a different Browser print(“n=== Study 1: constrained single-objective Bayesian optimization ===”) c1 = Client() c1.configure_experiment(parameters=SEARCH_SPACE, name=”rf_constrained”) c1.configure_optimization(objective=”accuracy”, outcome_constraints=[“model_size <= 2500”]) rec1 = run_study(c1, total_trials=24, metric_keys=[“accuracy”, “model_size”]) best_params, prediction, best_idx, best_arm = c1.get_best_parameterization() print(“nBest feasible configuration found:”) for k, v in best_params.items(): print(f” {k:>16}: {v}”) print(” predicted:”, prediction) feasible = [(r[“trial”], r[“accuracy”]) for r in rec1 if r[“model_size”] <= 2500] best_so_far, cur = [], -np.inf for _, acc in feasible: cur = max(cur, acc); best_so_far.append(cur) plt.figure(figsize=(7, 4)) plt.plot(range(1, len(best_so_far) + 1), best_so_far, “o-“) plt.xlabel(“feasible trial #”); plt.ylabel(“best accuracy so far”) plt.title(“Study 1 — convergence (subject to model_size <= 2500)”) plt.grid(alpha=0.3); plt.tight_layout(); plt.show() We run a constrained single-objective Bayesian optimization study where we maximize accuracy while keeping model size below a fixed threshold. We use Ax to suggest hyperparameter configurations, evaluate them, and report both accuracy and model size back to the optimizer. We then extract the best feasible configuration and plot the best accuracy achieved over feasible trials. Copy CodeCopiedUse a different Browser print(“n=== Study 2: multi-objective (accuracy vs. model_size) ===”) c2 = Client() c2.configure_experiment(parameters=SEARCH_SPACE, name=”rf_multiobjective”) c2.configure_optimization(objective=”accuracy, -model_size”) rec2 = run_study(c2, total_trials=28, metric_keys=[“accuracy”, “model_size”]) try: frontier = c2.get_pareto_frontier() print(f”Ax identified {len(frontier)} Pareto-optimal configurations.”) except Exception as e: frontier = None print(“get_pareto_frontier unavailable in this version:”, e) acc = np.array([r[“accuracy”] for r in rec2]) size = np.array([r[“model_size”] for r in rec2]) order = np.argsort(size) pareto_idx, best_acc = [], -np.inf for i in order: if acc[i] > best_acc: best_acc = acc[i]; pareto_idx.append(i) plt.figure(figsize=(7, 5)) plt.scatter(size, acc, c=”lightgray”, label=”all trials”) plt.scatter(size[pareto_idx], acc[pareto_idx], c=”crimson”, zorder=3, label=”Pareto front”) plt.plot(size[pareto_idx], acc[pareto_idx], “–“, c=”crimson”, alpha=0.6) plt.xlabel(“model_size (lower = cheaper)”); plt.ylabel(“accuracy (higher = better)”) plt.title(“Study 2 — accuracy vs. model size trade-off”) plt.legend(); plt.grid(alpha=0.3); plt.tight_layout(); plt.show() We move from single-objective optimization to multi-objective optimization by jointly maximizing accuracy and minimizing model size. We use Ax to search for configurations that represent strong trade-offs between predictive performance and computational footprint. We then calculate and visualize the empirical Pareto frontier to understand how accuracy varies with model size. Copy CodeCopiedUse a different Browser print(“n=== Study 3: parameter constraints on a synthetic surface ===”) c3 = Client() c3.configure_experiment( parameters=[ RangeParameterConfig(name=”x1″, bounds=(0.0, 1.0), parameter_type=”float”), RangeParameterConfig(name=”x2″, bounds=(0.0, 1.0), parameter_type=”float”), ], parameter_constraints=[“x1 + x2 <= 1.5″], name=”constrained_surface”, ) c3.configure_optimization(objective=”-dist”) for _ in range(14): for idx, p in c3.get_next_trials(max_trials=1).items(): dist = (p[“x1”] – 0.9) ** 2 + (p[“x2”] – 0.9) ** 2 c3.complete_trial(trial_index=idx, raw_data={“dist”: float(dist)}) bp, _, _, _ = c3.get_best_parameterization() print(f”Best point: x1={bp[‘x1’]:.3f}, x2={bp[‘x2’]:.3f}, ” f”sum={bp[‘x1’] + bp[‘x2’]:.3f} (constraint: <= 1.5)”) print(“Unconstrained optimum would be (0.9, 0.9); Ax respects the boundary.”) We demonstrate parameter constraints using a simple two-dimensional synthetic optimization problem. We ask Ax to minimize the distance to a target point while enforcing the input constraint that the sum of the two variables remains below a boundary. We observe that the optimizer respects the constraint and finds the best feasible point near the constrained optimum. Copy CodeCopiedUse a different Browser print(“n=== Ax built-in analyses for Study 1 ===”) try: import plotly.io as pio if “google.colab” in sys.modules: pio.renderers.default = “colab” cards = c1.compute_analyses(display=True) print(f”Computed {len(cards)} analysis cards.”) except Exception as e: print(“Interactive analyses didn’t render in this environment:”, e) print(“(The matplotlib plots above already capture the key results.)”) print(“n=== Saving / loading the experiment ===”) try: c1.save_to_json_file(“ax_study1.json”) reloaded = Client.load_from_json_file(“ax_study1.json”) print(“Saved to ax_study1.json and reloaded successfully.”) rp, _, _, _ = reloaded.get_best_parameterization() print(“Best params from reloaded client match:”, rp == best_params) except Exception as e: print(“JSON persistence API differs in this version:”, e) print(“See: https://ax.dev/docs/recipes/experiment-to-json”) print(“nDone. You optimized a mixed-type search space with constraints, ” “traced a Pareto frontier, and persisted in the experiment.”) We use Ax’s built-in