Uncategorized

Anthropic has published a new plugin evals workflow for Claude Code. The claude plugin eval command runs a plugin against realistic prompts, grades what Claude produced, and compares the result with a run where the plugin is not loaded. It answers 3 questions plugin developers could not previously measure: does the skill trigger, does it survive an edit or a new model, and does it beat a bare model. Deployable: Yes. It runs on Claude Code v2.1.269 or later against any directory with a plugin.json or .claude-plugin/plugin.json manifest, or a skills-directory plugin. Every eval run and judge grader is a real model call billed to your plan or API account. What a case looks like An eval suite lives in an evals/ directory inside the plugin. Each case is a subdirectory holding a prompt.md and a graders/ folder. The prompt body goes to Claude exactly as written, and @path mentions are not expanded. Frontmatter on prompt.md can set max_turns (default 10), timeout_seconds (default 300), model, tags, and allowed_tools. Graders are markdown files whose frontmatter sets a type, an optional weight, and an optional arm. There are 6 types. Four cost nothing because they are computed from the transcript and the files on disk: regex, tool_used, tool_order, and file_exists. Two call a judge model and add to the bill: llm, which scores the reply against prose criteria you write, and baseline, which compares it against a reference answer. claude plugin eval init reads the plugin, asks what a good result looks like, proposes cases and graders, tries them, and writes the files. In CI, –bare <name> writes a blank template instead. The number that matters is Δ By default every case runs twice: a with-arm where the plugin is loaded and a without-arm where it is not. Their difference, Δ, is what the plugin contributed. If a case scores 1.0 in both arms, the plugin is not why it passed. The docs example output shows a single case at WITH 1.00, W/OUT 0.33, Δ +0.67 across 6 runs, costing an estimated $0.41 and taking 74 seconds. A grader marked with-only, typically tool_used: Skill, is reported as an indicator and excluded from the score, since the without-arm has no skill to fire. Anthropic calls out the most common first finding: a Δ near zero with the tool_used: Skill grader failing, which means Claude is not choosing the skill on natural phrasing. That is the defect claude plugin validate cannot see, because it checks manifest syntax and schema rather than behavior. Results land under evals/results/<timestamp>/report.html with per-grader verdicts and judge votes. Where the account supports it, the report is also published to claude.ai unless –no-publish is set. Cost and CI A suite makes roughly cases × runs × arms agent runs, plus 3 short judge calls per llm or baseline grader per run, and results vary between runs. The documented CI invocation is: Copy CodeCopiedUse a different Browser claude plugin eval . –trust-plugin –json results.json –threshold 0.8 –model claude-sonnet-5 –judge-model claude-haiku-4-5 –no-publish –max-cost-usd 20 The runner needs a Claude Code install and credentials such as ANTHROPIC_API_KEY. Without –trust-plugin, an untrusted checkout is refused with exit 1 when there is no terminal. Report problems never change the exit code, and –json suppresses progress output. Interactive explainer Key Takeaways claude plugin eval scores realistic prompts with 6 grader types; 4 are free, llm and baseline bill a judge model. Every case runs with and without the plugin by default; Δ is the only number that proves the plugin did the work. A Δ near zero with a failing tool_used: Skill grader means the skill never triggers on natural phrasing. –threshold, –max-cost-usd, and –trust-plugin turn it into a CI gate; usage-limit errors can fake a regression. Requires Claude Code v2.1.269+; claude plugin eval init writes the first suite for you. Check out the Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills appeared first on MarkTechPost.

An agent harness is the code around a model: execution loop, tools, context, state, recovery, and verification. Per the Terminal-Bench 2.1 leaderboard, GPT-5 solves 35.2% of tasks inside Terminus 2 but 49.6% inside Codex CLI with identical weights. Most benchmarks keep that harness fixed. HarnessDev proposed by team of researchers from ByteDance Seed, Singapore University of Technology and Design, Georgia Institute of Technology, M-A-P, and TokenWave.AI, flips the target: the artifact under evaluation is the runnable harness the model writes, not the answer it produces. 2 stages: Creation and Evolution In Creation, every creator receives the same weak seed: passive file, search, and process primitives plus result and trajectory writers, with no loop, planner, verifier, retry, or stopping rule. Unmodified, it scores 0 everywhere. The creator gets a task-family spec, a short design tutorial, and 1 to 3 development cases, builds a full harness, and the harness is frozen before hidden tasks. In Evolution, the creator starts from its own frozen Creation code harness and revises it using execution feedback from a fixed set of 100 SWE-bench Pro tasks and all 89 Terminal-Bench 2.1 tasks. Each official candidate must complete both evaluations as a pair, with a budget of 10 pairs and at most 2 five-task probes between pairs. Every official version is later scored on 630 held-out SWE-Pro instances the creator never sees. Harnesses are graded on capability (task success) and efficiency (executor tokens, with creator tokens excluded). Setup 6 creator LLMs were tested: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro, working inside Claude Code 2.1.177 (GPT-5.5 used Codex 0.144.3). Creation spans 4 domains and 5 benchmarks totaling 2,207 instances: SWE-bench Pro public split (731), Terminal-Bench 2.1 (89), MLE-bench (75), EQ-Bench3 (46), and BrowseComp (1,266). Each creator builds 3 harnesses per benchmark, reported as avg@3. Self-Eval runs each harness with its creator; Unified-Eval runs all with Gemini 3.1 Pro. Creation results Under Self-Eval, Opus 4.8 posts the highest average score at 67.8 against a human-engineered reference of 86.2. The gap depends on domain: Code: Opus 4.8 reaches 69.3 on SWE-Pro versus the 80.0 reference. Gemini 3.1 Pro leads Terminal-Bench at 68.8 versus 88.8. Search: the widest gap. The best BrowseComp score is 52.6 (GPT-5.5) against a 92.2 reference. Writing: Opus 4.8 scores 84.6 on EQ-Bench3, above the 83.7 reference. ML experimentation: Opus 4.8 (32.9) and Gemini (32.4) beat the 24.0 MLE-bench reference. The SWE-Pro, Terminal-Bench, and BrowseComp references are external results from OpenAI’s GPT-5.6 report, not re-runs. Code volume did not predict quality: the 18 code harnesses added 17,111 net lines, yet Gemini added the fewest (1,006) and led Terminal-Bench. Self-test count barely correlated with score (Spearman 0.13 to 0.26); revision calls reached 0.57. Much generated machinery is inert. Of 108 code component instances, 72 trigger in real runs and 18 never fire, all of them state and memory. 11 of 18 harnesses define a State class, yet no checkpoint event appears across 26,679 trajectories. 124 of 587 writing features are dead code. Cost and executor transfer MLE-bench token use varied roughly 19-fold. GPT-5.5 hit a 19.1 medal rate with 29.3M tokens while DeepSeek V4 hit 19.6 with 208.4M. Swapping the executor to Gemini reshuffled rankings: Qwen gained 17.6 points on BrowseComp and 12.9 on MLE-bench, while Opus 4.8’s SWE-Pro score fell from 69.3 to 33.0, partly because one harness hard-coded a 120-step limit around its original executor. The Opus search harness’s duplicate-query rate jumped from 10.1% to 88.2% after the switch. Evolution results 9 lineages (5 self-runtime, 4 fixed-Gemini) produced 73 official versions and 64 adjacent switches. All 5 self-runtime creators improved on held-out tasks, from +1.43 to +4.44 points (mean +3.11). Under fixed Gemini, only Opus improved; GPT-5.5 regressed 10.32 points. Progress was not monotonic. Of 64 switches, 8 regressed on both benchmarks, 16 on one, 27 gained only within the noise band, and 2 showed clear positive evidence. A single commit can vary by about ±4.75 pair-score points. Feedback and held-out scores moved in the same direction only 34 of 64 times (53.1%), and only 2 of 9 declared final versions were held-out optimal. Of 169 new functions or classes, 25 have no caller. The clearest win: Opus 4.8 noticed 99 of 100 runs reported success while only 48 passed, traced it to premature completion, and added a completion gate. Failure diagnosis was otherwise the weakest step: the dedicated trajectory interface was called only twice. Interactive explainer Key Takeaways HarnessDev scores the harness a model builds, not the answer it returns. Self-built harnesses match or beat references on writing and ML experimentation but trail badly on code and search. Harness quality is executor-specific; Opus 4.8 drops from 69.3 to 33.0 on SWE-Pro under Gemini. Evolution gains are small, noisy, and only 34 of 64 changes point the same way on held-out tasks. Much generated state and memory code never executes. Check out the Paper and Project Page. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize appeared first on MarkTechPost.

Biomolecular structure prediction has shifted from single-target runs to proteome-scale worklists. The bottleneck is no longer whether a model can fold a protein. It is how fast an entire queue of independent targets moves through parsing, featurization, GPU inference, and output writing. NVIDIA’s new technical deep dive walks through BioNeMo Inference Runtime (BioIR), a Python library that accelerates supported structure-prediction models on NVIDIA GPUs while keeping the standard PyTorch workflow. BioIR has already run at production scale. It powered the recent expansion of the AlphaFold Database, generating protein-complex structures across 4,777 proteomes, about 31 million candidate complexes, with 1.81 million released as high-confidence predictions. Is it deployable? Yes. BioIR is available now as an open GitHub repository with a wheel containing precompiled CUBINs. Runtime use needs Python 3.12+, a compatible NVIDIA GPU and driver, a staged model checkpoint, and per-chain A3M MSAs. It does not require nvcc, CUDA source, CMake, or the CUDA toolkit. What is BioIR BioIR targets the operations that general-purpose inference stacks do not fully optimize. These include Pairformer and Evoformer stacks, triangle operations, pairwise attention, diffusion transformers, and atom-level modules. Models stay ordinary torch.nn.Module objects. There is no engine build, export step, or separate artifact between a checkpoint and a forward pass. There are 2 ways to use it. The end-to-end processor moves an InputRequest through parsing, tokenization, feature generation, GPU inference, and PDB or mmCIF writing. Direct PyTorch integration lets developers construct a supported model or reuse selected optimized modules inside custom code. The tutorial demonstrates the processor path with Boltz-2 (model_source=”boltz-2″). Each protein chain requires an A3M MSA. Paired or unpaired MSAs are accepted for inputs with multiple non-identical protein chains. Templates can be supplied manually because BioIR does not run HHsearch or HMMsearch. The processor supports ligand structure prediction but not ligand-affinity prediction. Three Layers of Acceleration BioIR optimizes at 3 distinct layers, each targeting a different bottleneck: Kernel selection: Supported operations pick compatible BioIR custom, cuEquivariance, or PyTorch fallback implementations based on model configuration, GPU, data type, and tensor shape. Module optimization: A separate optimize() mechanism enables CUDA Graph capture for compatible modules, cutting launch overhead. Pipeline scaling: A Ray executor places 1 complete model replica on each visible GPU in a node and distributes independent inputs among them. CPU stages (parsing, featurization, writing) overlap with GPU folding. Note: Ray does not split a single forward pass across GPUs. Replica mode scales worklists, not individual targets. Per the support matrix, context-parallel folding is planned but not yet available. The capacity rule is simple: engine_stage.compute x num_gpus must not exceed visible GPUs. At the model-forward level, NVIDIA’s early benchmarking reports geometric-mean speedups over an OSS torch.compile baseline of 1.55x (OpenFold3), 1.78x (Boltz2), and 2.56x (OpenFold2 monomer) on H100. H200 numbers are similar at 1.54x, 1.75x, and 2.61x. These were measured across 17 inputs spanning 29 to 1,734 residues. The Benchmark: 1,000 Human Dimers on 8xH100 To quantify end-to-end delivery, NVIDIA team ran a matched benchmark on 1,000 human dimer targets with combined sequence lengths below 2,800 residues. The comparison pitted BioIR-accelerated Boltz-2 against a torch-compiled open-source Boltz-2 implementation on 8xH100 80GB GPUs. Both used identical targets, staged MSAs, inference recipe (3 recycles, 200 sampling steps, 5 diffusion samples), and GPU configuration. The results: BioIR completed all 1,000 targets and delivered 58.5K successfully folded residues per allocated GPU-hour. The public implementation delivered 20.2K residues per GPU-hour and ran out of memory on 29 targets. Net result: a 2.90x improvement in residue-normalized throughput. These numbers are folding-stage measurements specific to this dataset and hardware. They exclude MSA generation, preprocessing CPU allocations, storage, data transfer, and retries. The blog explicitly warns against generalizing them to all BioIR-supported models or datasets. Energy at One Million Targets Extrapolating the benchmark linearly to 1 million comparable targets, BioIR is estimated to need 11 MWh versus 35 MWh for the public implementation using 8-GPU TDP equivalents. Using full-node maximum-power equivalents, the estimate is 21 MWh versus 64 MWh. These are rated-power, folding-only estimates for IT equipment, not metered measurements, and exclude data center overhead such as PUE. Still, a 23 to 43 MWh saving per million targets is a material number for proteome-scale campaigns. Key Takeaways BioIR accelerates Boltz-2, OpenFold2, and OpenFold3 inference on NVIDIA GPUs while staying in plain PyTorch. Matched 8xH100 benchmark: 58.5K vs 20.2K folded residues per GPU-hour, a 2.90x throughput gain. Ray replica mode scales independent worklists; it never splits 1 forward pass across GPUs. Estimated energy for 1M targets drops from 35 MWh to 11 MWh at 8-GPU TDP equivalents. Already proven at scale: 31M candidate complexes generated for the AlphaFold Database expansion. Check out the technical blog, GitHub repo, docs, and the BioNeMo Agent Toolkit for agentic orchestration. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post NVIDIA Details BioNeMo Inference Runtime (BioIR): 2.90x Higher Boltz-2 Folding Throughput and 58.5K Residues per GPU-Hour on 8xH100 appeared first on MarkTechPost.

OpenAI has released the Agents API in public beta. It gives developers the same harness and infrastructure that run Codex. OpenAI hosts and maintains the harness. Developers run the agent’s compute in an OpenAI-managed sandbox, their own infrastructure, or a partner sandbox. Is it deployable? Yes. It is live for all developers in public beta. Data stays US-only, and Zero Data Retention is unsupported. What OpenAI Shipped The Agents API is a managed service built on the open-source Codex harness. OpenAI team states scaling Codex and ChatGPT for Work showed what long-running agents need. They need a harness that manages context, uses tools efficiently, and coordinates subagents. They also need infrastructure that keeps them running reliably for days. The official docs organize the API around 4 concepts: Agent: the model, instructions, tools, and MCP servers available to it. Environment: an optional sandbox where the agent accesses files, loads skills, and runs commands. Session: a durable agent instance that works on tasks and responds to input. Events and items: the inputs sent to the agent and the output it produces. A session runs in 4 steps. You create it and give it a task. Then you follow progress through streaming or webhooks. Finally, you continue with a new task or steer the current turn. One API Call OpenAI’s announcement shows an incident-investigation agent created in a single call: Copy CodeCopiedUse a different Browser import OpenAI from “openai”; const client = new OpenAI(); const session = await client.beta.agents.sessions.create({ agent: { model: “gpt-6-astra”, tools: [ { type: “mcp”, server_label: “observability”, transport: { type: “http”, server_url: “https://observability.example.com/mcp”, }, }, ], multi_agent: { enabled: true, max_concurrent_subagents: 3 }, }, vault_ids: [“vault_YOUR_VAULT_ID”], environment: { type: “openai_hosted”, capability_directories: [“/workspace/capabilities/skills”], }, input: “Investigate service-api’s elevated 5xx rate over the last 30 minutes. ” + “Delegate deployment, error, and dependency analysis to subagents. ” + “Save findings, evidence, and recommended mitigation in /workspace/outputs.”, }); The quickstart covers API key permissions and SDK setup. Where the Agent Runs Environment choice is the main architectural decision. The Agents API supports 3 sandbox options, and it can also run without a sandbox. OpenAI-hosted sandbox: uses the sandboxing infrastructure behind Codex and ChatGPT. You can configure it with files, packages, skills, and plugins. Self-hosted: you run codex exec-server inside your environment. It registers with a restricted key and connects over WebSocket. All connections are outbound. Partner sandboxes: Blaxel, Cloudflare, Daytona, DigitalOcean, E2B, Modal, Oracle, Runloop, and Vercel have first-class integrations. What the Harness Handles OpenAI maintains the harness alongside its models, with versioned access at each model launch. Long sessions: The API automatically compacts earlier context as a session nears its limit. Developers do not write their own compaction logic. Efficient tool use: Tool search loads tool definitions only when needed. This reduces token usage and cost while preserving the model’s cache. Programmatic tool calling lets agents run calls in parallel and chain operations. Agents filter or combine results in code, so only relevant data returns into context. Supported tools include MCP, custom functions, and built-in tools like web search. Subagents: With multi-agent support, the main agent splits complex tasks into independent pieces. Each subagent keeps its own context. The main agent coordinates them and combines the results. Agents API vs Agents SDK vs Responses API OpenAI’s runtime comparison positions the 3 options this way: Agents API Agents SDK Responses API Where the agent runs OpenAI runs a managed Codex harness Inside your application Your application, with optional hosted orchestration Integration effort Low Medium High State between tasks Saved session configuration, turns, and items Your storage and SDK sessions Manual history, response chaining, or Conversations Execution environment OpenAI-hosted, self-hosted, or no sandbox Your runtime and sandbox providers Your own environment Early Customer Results OpenAI published these customer-reported numbers. They are vendor-supplied, not independent benchmarks. Ciridae: evaluation score rose from 0.71 to 0.85, with a 4x latency reduction on subagent flows. SafetyKit: 60% lower cost per case after migrating its case review workflow. Hypha: 86% fewer failed agent responses after separating the harness from the sandbox. Nash.ai: runs thousands of long-running agents across global logistics networks. Key Takeaways OpenAI’s Agents API exposes the managed Codex harness as a public beta API. Agents run in OpenAI-hosted, self-hosted, or 9 partner sandboxes. Compaction, tool search, programmatic tool calling, and subagents come built in. There is no extra fee; you pay for tokens, tools, and container time. US-only data residency and no ZDR limit regulated workloads for now. Check out the Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post OpenAI Launches the Agents API in Public Beta, Putting the Codex Harness Behind One API Call appeared first on MarkTechPost.

Production LLM applications rarely receive a question nobody has asked before. Support assistants and RAG pipelines field the same intents thousands of times a day, each phrased differently, and most stacks treat every phrasing as a fresh, fully billed request. Redis LangCache is a fully managed semantic caching service that sits between the application and the model, matches incoming prompts against previously answered ones by meaning rather than exact text, and returns the stored response when a close enough match exists. Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster than re-querying the model. Is it deployable? Yes. LangCache is available today as a public preview on Redis Cloud, accessed through a REST API with Python and JavaScript SDKs, and Redis notes that features and behavior may change during the preview. The Problem: Paraphrases Are Still Full LLM Calls Consider three requests to a customer-support assistant: “Can I get a refund after buying the monthly plan?” “Is the monthly subscription refundable?” “Can I cancel the plan and get my money back?” The wording differs, but the question and answer are identical. Without a semantic cache, each version triggers a complete generation: input tokens processed, output tokens decoded, user waiting. Prefix caching only removes part of that cost. When requests share a system prompt or context, the engine reuses the KV states computed for that prefix, but the request still reaches the LLM, new tokens still get processed, and the full answer still gets decoded. A prefix-cache hit is a cheaper generation call, not an avoided one. How LangCache Works LangCache moves the cache outside the model and stores the generated response itself. The architecture is a two-call loop: Before invoking the model, the app sends the prompt to POST /v1/caches/{cacheId}/entries/search. LangCache generates an embedding for the prompt and runs a vector search over stored entries. If a semantically similar entry clears the configured similarity threshold, the cached response is returned and no LLM call occurs. On a miss, the app calls its chosen LLM as usual, then stores the prompt and new response through POST /v1/caches/{cacheId}/entries for future matches. Embedding generation is handled by the service, with default models or bring-your-own. Cache behavior is controlled through similarity thresholds, TTLs, and eviction policies, plus adaptive controls that tune precision and recall. Built on Redis’s vector database and exposed as a REST API, it works with any LLM provider and language. Hit rates and savings are monitored from the Redis Cloud console. What a Cache Hit Actually Saves A cache hit removes the input tokens, the output tokens, and the decoding latency of an additional model call. In a demo run comparing both paths on a paraphrased question, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens. LangCache returned the earlier response in 0.37 seconds with zero LLM input or output tokens, roughly 6x faster in that run. The Redis documentation is careful about how savings accrue. On a cached response you do not pay for output tokens, while input token costs are typically offset by embedding and storage costs. The suggested estimate is: Est. monthly savings = (Monthly output token costs) x (Cache hit rate) With $200 of monthly LLM spend, 60% of it on output tokens, and a 50% hit rate, that works out to $60 saved per month. Redis also publishes a savings calculator for annual estimates. Redis’s public preview announcement cited up to 15x faster responses on cache hits and up to 70% lower token usage, while the current product page states savings of up to 90%. Customer Mangoes.ai reports a 70% hit rate on its patient-care voice app, cutting LLM spend by 70% with 4x faster responses. The actual result depends on how much safe repetition exists in the traffic. Where Semantic Caching Needs Care Deciding which questions can safely share an answer is a production concern, not a configuration detail. A threshold set too low returns a refund policy to a customer asking about upgrades. Set too high, nearly every paraphrase goes back to the model and the cache stops paying for itself. Production setups need well-tuned thresholds, expiration policies so stale answers age out, data isolation between tenants, and monitoring for incorrect matches. LangCache covers these with access scopes, custom filtering, TTL and eviction controls, and monitoring through Redis Cloud. Data stays on the customer’s Redis servers, and Redis states it does not access that data or use it to train models. Key Takeaways Prefix caching cuts prompt-processing cost; semantic caching eliminates the LLM call entirely on a hit. LangCache is a two-call REST integration: search before the model, store after it. Savings come mainly from avoided output tokens; the docs give the formula output cost x hit rate. Redis claims up to 90% cost savings and up to 15x faster cache hits; a demo run showed 6x. Thresholds, TTLs, isolation, and false-match monitoring decide whether a semantic cache is safe. Check out redis.io/langcache and follow the API and SDK examples. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Meet Redis LangCache: A Managed Semantic Cache That Cuts LLM API Costs by Up to 90% and Returns Cache Hits Up to 15x Faster appeared first on MarkTechPost.