Committee

A team of researchers from UC Berkeley have released CUA-Lite, an open platform for computer-use agents (CUAs). The argument behind it is infrastructural rather than model-centric: training and benchmarking a CUA requires four pieces: agents, environments, traces, and a framework to evaluate and train them and all four are currently fragmented across separate repositories with incompatible interfaces. CUA-Lite puts them behind one action space, one data schema, and one command, across desktop, browser and mobile. Is it deployable? Yes. The stack installs with uv sync –all-extras on Python 3.12, and its lightweight sandboxes run on any Docker host without /dev/kvm, so cloud instances, CI runners and nested containers all work. The VM tax, and how Lite.OSWorld removes it The most concrete contribution is Lite.OSWorld. OSWorld provides a faithful Ubuntu desktop, but it ships as a full QEMU/KVM virtual machine per task, requiring nested virtualization that most managed infrastructure does not expose. CUA-Lite reproduces the same task suite and the same evaluators on a GNOME desktop inside a plain Docker container. Task OSWorld Lite.OSWorld Runtime QEMU/KVM VM Docker container Host requirement /dev/kvm, nested virt Any Docker host Memory 4.1 GB 0.9 GB Cold start 29.9 s 23.8 s Parallelism baseline ~4.6× more instances Task suite OSWorld Identical Fidelity is the obvious concern when you swap a VM for a container, and the team addresses it directly: across 13 models, Lite.OSWorld scores match the OSWorld VM’s, so a score or a training signal earned in the container transfers back to the real benchmark. The same base now carries a family of sandboxes: Lite.ScaleCUA, Lite.CUAGym and Lite.CUAWorld, the last expanding into roughly 40 applications including Blender, QGIS and VS Code. In total the platform claims 30k+ verifiable tasks. One schema for data, one adapter per model CUA-Lite’s second layer is LiteSample, a single supervised-learning schema shared across every environment, agent and task type, shipped as plain parquet plus images. Ten-plus existing CUA datasets have been preprocessed into it and published free on Hugging Face, including Aguvis, OpenCUA, ScaleCUA, GUI-360, GUIOdyssey and Multimodal-Mind2Web. Alongside those corpora sit fresh rollout datasets generated by rolling a frontier teacher model through the sandboxes, for distillation into smaller students. Because model families expect different scaffolding, the framework ships a per-model adapter that packs a unified LiteSample into each model’s own training format, including history collapsing so several steps share one forward pass. Eval, SFT and RL behind one command Agents and environments meet in lite.gym: screenshots up, actions down, with one action space per platform. 10+ agents are built in GPT, Claude, Gemini, Qwen3-VL, UI-TARS, Fara-7B, MAI-UI and others, and 15+ benchmarks are integrated, spanning grounding (ScreenSpot-Pro, OSWorld-G), desktop (OSWorld, OSWorld-2, WindowsAgentArena, CUABench), browser (WebArena, VisualWebArena, MiniWoB, WebVoyager, Online-Mind2Web, WebGym) and mobile (AndroidWorld, AndroidLab, MobileWorld, MobileGym). Swapping –model-id and –env-id in scripts/rollout.py is the whole interface. The same loop serves training. For SFT, the README documents fine-tuning Qwen3-VL-2B-Instruct on Lite.ScaleCUA desktop trajectories, lifting mean episode return from 0.138 to 0.237 on the 332-task lite.osworld eval split, a single reported configuration on two GPUs, not an independently reproduced result. For RL, rollouts scored in the environment drive GRPO updates on top of Slime, with a worked MobileGym example covering 416 mobile tasks across 28 apps. Interactive explainer Key Takeaways CUA-Lite unifies agents, environments, traces and training under one action space and one LiteSample schema. Lite.OSWorld runs OSWorld tasks VM-free in Docker at 0.9 GB versus 4.1 GB, roughly 4.6× more parallel desktops. Scores in the container match the OSWorld VM across 13 models, so training signal transfers to the real benchmark. 30k+ verifiable tasks, 15+ benchmarks, 10+ agents, and 20+ datasets published free on Hugging Face. Deployable on any Docker host, but the repository ships no explicit license yet — verify terms before commercial use. Check out the Project Page, GitHub Repo and Datasets on Hugging Face. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post UC Berkeley Researchers Release CUA-Lite, an Open Platform Unifying Sandboxes, Data, Evaluation and RL for Computer-Use Agents appeared first on MarkTechPost.

Retrieval quality in an AI search product is bounded by two things: how good the embedding model is, and how cheaply you can run it across an index. This week, Perplexity Engineering team published Fast Embeddings on GPUs, an under-the-hood account of the second — the serving infrastructure behind pplx-embed and the ranking models used across Perplexity Search, Computer and the API Platform. Perplexity team states that embedding inference on the GPU side has largely converged across engines on mature Hopper and Blackwell hardware. The wins sit in the runtime and harness around the model: CUDA graph management, an async result-tracking abstraction, and a Rust request path. Two traffic patterns, one engine Perplexity frames embedding serving as two workloads. Batch embedding happens when building or re-indexing the vector database, where throughput minimizes cost. Online embedding happens at query time, where a short query must be embedded fast. Scoring sits in between: after vector search, large document batches are ranked, balancing both. The key decision is that Perplexity did not build a separate embedding engine. Because embedding models are small Transformers, batch embedding resembles compute-bound prefill and online embedding, often a few tokens, resembles memory-bound decode. So the research team reuses the prefill and decode kernels from its LLM stack. Ivy, Tulip and ROSE Three services handle a request: Ivy is a Rust HTTP gateway. It does the CPU-side work — JSON parsing, tokenization, input templating, batch splitting — and translates requests into a custom gRPC protocol. It also splits large-batch requests into chunks and load-balances them across replicas, which corrects the load imbalance that arises when production payloads vary in size. Tulip is the inference server interface: a gRPC server built with Rust, tokio and tonic, handling scheduling and batching before dispatching to the engine. ROSE (Runtime-Optimized Serving Engine) implements model inference. It is primarily Python, provides kernels, layers and model definitions, manages CUDA graphs, and exposes a step() function to Tulip. Why the scheduler is deliberately simple Tulip picks sequences first-come, first-served while requests accumulate. That simplicity is justified by a measurement: for small embedding models at the sequence lengths Perplexity serves, the linear cost of dense layers dominates the quadratic cost of attention. Latency is therefore roughly proportional to token count, not sequence count. Once a batch saturates the GPU, around 512 tokens on a sub-billion-parameter model, packing in more sequences does not improve efficiency. CUDA graphs and LazyTensors On small batches, CPU-side kernel launching can outweigh GPU execution. Perplexity builds whole-model CUDA graphs for all embedding models, capturing every launch into a single driver call. Because embedding models are small, the inflection point where GPU work exceeds launch cost arrives at batches of thousands of tokens and tens of sequences. Some attention implementations block full-model graphs by depending on dynamic host-side inputs; Perplexity upstreamed changes to FlashInfer to enable capture. Graphs must be captured per configuration, so token counts are padded to buckets that are multiples of 64 or 256. That still yields thousands of graphs and multiple minutes of capture per model. The fix is lazy capture: each configuration gets an eager warmup run, then triggers capture and replay on its second hit. This costs p99 latency at startup but spreads minutes of eager work across hours. The second piece is the LazyTensor, which tracks a page-locked host buffer plus a cudaMemcpyAsync and a CUDA event. Instead of step() blocking on the device, it returns a LazyTensor, letting a Rust async task wait on batch N while the CPU enqueues N+1. Send a request</button></div> </div> </div> <div class=”pe-panel” id=”peP1″> <div class=”pe-card”> <div class=”pe-txt”>On small batches, CPU-side kernel launches can outweigh GPU work. A whole-model <b>CUDA graph</b> captures every launch into one call to the driver, so the CPU is freed to enqueue the next batch. Toggle the two modes.</div> <div class=”pe-ctl” style=”margin:0 0 12px”> <button class=”pe-btn ghost on” id=”peEager”>Eager launches</button> <button class=”pe-btn ghost” id=”peGraph”>CUDA graph</button> </div> <div class=”pe-lane”><div class=”pe-lbl”>Host / CPU</div><div class=”pe-track” id=”peCpuT”></div></div> <div class=”pe-lane”><div class=”pe-lbl”>Device / GPU</div><div class=”pe-track” id=”peGpuT”></div></div> <div class=”pe-stats”> <div class=”pe-stat”><div class=”v” id=”peLaunches”>—</div><div class=”k”>Driver calls</div></div> <div class=”pe-stat”><div class=”v” id=”peGap”>—</div><div class=”k”>GPU idle gaps</div></div> </div> <div class=”pe-txt” style=”margin:12px 0 0;font-size:11.5px;color:#6E8285″>Schematic. Block widths illustrate the launch-overhead pattern described in the post, not measured timings.</div> </div> </div> <div class=”pe-panel” id=”peP2″> <div class=”pe-card”> <div class=”pe-txt”>Reading results back normally forces a host sync. A <b>LazyTensor</b> tracks a page-locked host buffer plus an async device-to-host copy and a CUDA event, so Tulip can block on batch N while the CPU already prepares batch N+1.</div> <div class=”pe-lane”><div class=”pe-lbl”>CPU — prepare / sync</div><div class=”pe-track” id=”peLzC”></div></div> <div class=”pe-lane”><div class=”pe-lbl”>GPU — forward pass</div><div class=”pe-track” id=”peLzG”></div></div> <div class=”pe-ctl”> <button class=”pe-btn” id=”peLzRun”> Run 3 batches</button> <button class=”pe-btn ghost on” id=”peLzOn”>Overlapped</button> <button class=”pe-btn ghost” id=”peLzOff”>Blocking</button> </div> <div class=”pe-note” style=”margin-top:12px” id=”peLzNote”>Overlapped: while the GPU chews batch N, the CPU is already tokenizing and packing batch N+1.</div> </div> </div> <div class=”pe-panel” id=”peP3″> <div class=”pe-card”> <div class=”pe-txt”>For small embedding models at these sequence lengths, the linear cost of dense layers dominates the quadratic cost of attention, so latency tracks <b>token count, not sequence count</b>. Past roughly <b>512 tokens</b> on a sub-1B model, the GPU is saturated and packing in more sequences stops helping.</div> <div class=”pe-lbl” style=”margin-top:6px”>Tokens in batch: <span id=”peTokV” style=”color:#3FB6C4″>512</span></div> <input type=”range” id=”peTok” min=”32″ max=”4096″ step=”32″ value=”512″> <div class=”pe-lbl”>GPU utilisation</div> <div class=”pe-bar”><div class=”pe-fill” id=”peUtil”></div></div> <div class=”pe-stats”> <div class=”pe-stat”><div class=”v” id=”peUtilV”>—</div><div class=”k”>Saturation</div></div> <div class=”pe-stat”><div class=”v” id=”peState”>—</div><div class=”k”>Regime</div></div> </div> <div class=”pe-txt” style=”margin:12px 0 0;font-size:11.5px;color:#6E8285″>Illustrative curve. The ~512-token saturation point is the figure stated in the post; the shape between points is a stand-in, not a benchmark.</div> </div> </div> <div class=”pe-foot”> <span>Source: Perplexity Engineering, “Fast Embeddings on GPUs” (Sep 4, 2026)</span> <span><a href=”https://www.marktechpost.com”>Built by Marktechpost</a></span> </div> <script> (function(){ var R=document.getElementById(‘pplxEmbedExplainer’); var NOTES=[ ‘<b>Ivy</b> — parses JSON, tokenizes with the in-house unigram tokenizer, applies input templating and splits large batches, then translates to a custom gRPC protocol. It also load-balances chunks across replicas.’, ‘<b>Tulip</b> — Rust gRPC server on tokio and tonic. Requests accumulate while it dispatches or waits; sequences are picked first-come, first-served and packed into a batch for the accelerator.’,

AI research agents can already propose, implement and score their own machine learning experiments. Idea generation is cheap; verification is not. Training one candidate can consume hours to days of GPU time, so an agent proposes far more candidates than it can afford to run. Which ones get run is the real lever on research progress. A research team from FAIR at Meta, the University of Oxford and University College London formalizes that lever as research preference and introduces AI Research Preference Models (RPMs). An RPM ranks unexecuted candidates and picks one to execute. It never forecasts an absolute score, the team found language models unreliable at predicting metrics or execution outcomes. Is it deployable? Partially. RPMs use frozen pretrained LLMs with no fine-tuning, the scaffold AIRA-dojo and benchmark AIRS-Bench are open source, and the backbone Qwen3.6-27B is open weights. Where the RPM sits in the agent loop AIRA-dojo is an evolutionary tree search: greedy parent selection, Draft / Improve / Debug operators, highest-validation-score node returned at the end. The RPM intervenes at child creation only. Instead of generating one child and executing it, the agent applies the operator 15 times in parallel to yield 15 unexecuted candidates, then compares them pairwise in a knockout tournament. Only the winner is executed. Each comparison is grounded in context nodes collected by a BFS walk of the explored tree, each shown with the validation score it obtained. Two variants, two compute budgets Inference-only RPM: An LLM-as-a-judge over candidate plans, code and search history. Its prompt was optimized with MIPROv2 from DSPy, converging on a principal-investigator rubric that tolerates fixable bugs, rewards extensibility and penalizes redundant directions, offline accuracy 57.7% to 59.0%. Agentic RPM: The same judge and a sandbox that clones the agent’s environment, including a single H200. Tools are python, bash and submit_solution. It runs small-scale pilot experiments, then a feedback model either proposes the most informative next experiment or ends the loop. Two design choices carry weight: the remaining budget is deliberately overstated (2,700s reported against a real 300s) so the agent does not stop early, and pilots are capped at 30 with a 60-second threshold. Pilot time competes with the agent’s own clock, so the agentic selector runs only on Draft and Improve steps; Debug reverts to random. Results on AIRS-Bench Setup: 20 public text and tabular tasks, 24 hours on a single H200 per task, 10 seeds, Qwen3.6-27B as backbone for both the operators and the RPM, so the gain comes from the selection layer, not a stronger judge. Child selection Avg. normalized score No RPM (random pick) 0.684 Inference-only RPM 0.711 Agentic RPM 0.729 Validation oracle (ceiling) 0.748 Test oracle (ceiling) 0.759 Probability of improvement over No-RPM is 0.5923 and 0.5913, with 95% CI lower bounds at 0.5066 and 0.5018. Efficiency is the more practical result. Inference-only reaches the baseline’s final 0.684 in 14.88 hours (1.61×), agentic in 15.50 hours (1.55×). Self-hosted inference adds 0.660 hours per run; adjusting for it still gives 0.708 at 23.34 hours. Two new reported SOTA results: WinoGrande 94.1% with the Agentic RPM against a prior agentic SOTA of 90.4% from AIRA₂, and SVAMP 95.7% with inference-only against a prior human SOTA of 94.2%. Key Takeaways RPMs rank unexecuted candidates so an AI research agent runs only the most promising one. Two frozen-LLM variants: an inference-only judge, and an agentic judge that runs short pilots. On AIRS-Bench, average normalized score rises from 0.684 to 0.711 and 0.729. Both hit the baseline’s 24-hour score in roughly 15 hours, a 1.5–1.6× speedup. New reported SOTA on WinoGrande (94.1%) and SVAMP (95.7%). Check out the Paper and the LinkedIn announcement. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Meta FAIR Introduces AI Research Preference Models (RPMs): Ranking ML Experiments Before Spending GPU Hours appeared first on MarkTechPost.

Multi-agent workflows have changed the shape of local inference. A lead agent decomposes a task and spawns subagents. What looked like one user request becomes dozens of independent model calls. Pointed at a single local engine, those calls compete for the same execution slots. The queue grows while a workstation, laptop, or DGX Spark on the same network sits idle. NVIDIA Personal AI Router (PAIR) targets exactly that bottleneck. Announced this week, PAIR is a virtual inference router. It discovers compatible machines on a home network and schedules independent inference requests across them. It is not a new inference engine. Ollama or LM Studio still executes the model on whichever node PAIR selects. Is it deployable? Yes. PAIR ships today as a public beta (v0.1.1) with signed installers for Windows, macOS, and Linux, and the full source is on GitHub under Apache 2.0. It runs entirely on the local network, with internet needed only to download models. No new API The design decision that matters most is that PAIR introduces no cluster API. It proxies the Ollama-compatible and LM Studio-compatible interfaces agents already speak, taking over the default port each engine uses. If a harness listens elsewhere, the proxy port is configurable in PAIR’s engine settings. The repository also exposes OpenAI-compatible proxy endpoints. The consequence: existing agent harnesses need no changes. The agent decides what work to request. PAIR decides where it runs. Discovery, pairing, and transport PAIR uses mDNS to find nearby systems automatically. A node can be added by IP address when discovery fails. Trust is established by a six-digit PIN shown on the inviting machine and entered on the invited one. All node-to-node communication is blocked until that pairing completes. Traffic between paired nodes is then secured with mTLS using generated certificates. Each node runs Ollama or LM Studio. PAIR can install an engine and start model downloads on paired systems, removing most cross-machine setup work. How the scheduler picks a node A node becomes eligible for a request only when the required engine is enabled and the exact requested model is present. Models do not need to be identical across the cluster, different systems can hold different models, and PAIR routes according to model location. Loading the same tag on more nodes simply widens the eligible pool. For each request the scheduler weighs five signals. Is the node online and ready. Is a supported engine enabled. Is the exact model present. What is the current node and engine job load. What is existing GPU utilization. This is workload-level concurrency, and the boundary is explicit. PAIR assigns each request to one eligible node, where it stays for its lifetime. It does not pool VRAM, merge GPUs into one larger accelerator, or shard a single request across machines. The demo numbers, with their caveats NVIDIA’s demonstration pairs PAIR with Hermes Desktop, which creates a five-subagent workload over a synthetic household inbox. Ollama executes Qwen 3.6 35B A3B on each selected node. On one RTX Spark laptop, the workload took 18 minutes on average. On a three-device PAIR cluster consisting of an RTX Spark laptop, a DGX Spark, and an RTX 5090, it averaged 8 minutes and 48 seconds. Supported hardware and requirements PAIR supports GeForce RTX 20 Series GPUs and newer, RTX PRO workstation GPUs from Turing onward, DGX Spark, and Apple M4 or newer silicon. Windows, Linux, and macOS nodes can all be paired with each other, on x64 and arm64, though Windows on ARM is experimental. Validated configurations list 8 GB RAM or higher and a recommended 20 GB of disk. Other Linux distributions build from source. Key Takeaways PAIR routes each independent request to one node; it never pools VRAM or shards a model. It proxies existing Ollama and LM Studio endpoints, so agent harnesses need no changes. Eligibility depends on node readiness, enabled engine, exact model presence, job load, and GPU utilization. NVIDIA’s five-subagent demo went from 18 minutes on one laptop to 8:48 on three devices, unofficially. Apache 2.0 and beta: one scheduling policy today, blind to VRAM, GPU class, and model warmness. Check out the NVIDIA PAIR product page, NVIDIA Technical Blog, GitHub repository, Playbook and FAQ. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post NVIDIA Releases Personal AI Router (PAIR): An Open Source Virtual Inference Router that Distributes Local AI Requests Across RTX, DGX Spark, and Mac Nodes appeared first on MarkTechPost.

This week, Adaption Labs released Invent a Dataset. The feature generates a structured, training-ready dataset from a description of the behavior you want a model to learn. You do not arrive with a seed corpus, a predefined schema, or a labeling guide. Is it deployable? Yes, with one caveat. Invent a Dataset is live now in the Adaption app and through the Python SDK and REST API. Generated rows download as JSONL, JSON, CSV, or Parquet, so the artifact is a portable file you own and can train on anywhere. Generation itself runs on Adaption’s hosted platform and consumes credits. No self-hosted generation path is documented. The problem being targeted Most dataset workflows begin with data that already exists. Teams then spend weeks labeling, filtering, and reshaping it to approximate the target task. Adaption’s argument is that this caps model quality at how closely the available data matches the intended behavior. For proprietary and specialized tasks, the relevant signal usually sits in internal systems, unstructured text, or workflow logs. It rarely converts cleanly into a focused training set. The research team also draws a line against existing synthetic-data tooling. Those tools automate generation after a human has already defined the schema, task distribution, and generation strategy. Invent a Dataset starts one level earlier, at the behavior itself. How the API works The mechanics are documented and concrete. A single call to datasets.invent creates the dataset and starts generation, returning immediately with status running. You then poll datasets.get until the status reads succeeded or failed, and download the rows. Domain codes are the primary control. You fetch current codes with datasets.invent_domains rather than hardcoding them. You then pass values such as medical, optionally narrowed by qualified subdomain codes such as medical.symptoms_diagnosis. At least one domain or subdomain is required. Multiple domains contribute to the same run. A domain passed without subdomains draws from its full scope. Two output formats are supported. instruction_dataset is the default and produces prompt-completion pairs for supervised fine-tuning. preference_pairs produces chosen and rejected completions for preference-based training such as DPO. Three parameters matter for production use. estimate=True prices the exact request and returns estimated versus available credits without creating or charging anything. prompt accepts up to 10,000 characters to steer what the rows are actually about. idempotency_key accepts up to 255 characters and makes network retries safe by returning the original dataset instead of launching a second run. Row counts are subject to a per-launch limit set by your plan. Language and locale expansion language_expansion runs in two modes. translate produces a new row variant for each target language. localize produces a variant for each country and language pair, using locale-specific wording rather than direct translation. A sample_rate between 0.01 and 1 controls what fraction of invented rows gets expanded, and credits are billed on the expanded output row count, not the original. Unsupported codes return a 400 with a sample of valid values. The zero-data loop Invent a Dataset is the first half of a loop. The dataset ID passes directly to autoscientist.create, which co-optimizes the data and the training recipe against your objective. AutoScientist launched in May 2026 and is the training-side counterpart to the Adaptive Data pillar. Adaption reports that AutoScientist beats training configured by its own research staff, by an average of 35%. Win rates moved from 48% to 64%. Those figures come from in-house domain-specialized evaluations across eight verticals. Dataset sizes ranged from 5,000 to 100,000 rows, on architectures offered for fine-tuning by Together AI. Key Takeaways Invent a Dataset generates training rows from a task description, with no seed corpus, schema, or labels. One datasets.invent call sets domains, row count, format, and language expansion; generation is async. Output is instruction pairs or preference pairs, downloadable as JSONL, JSON, CSV, or Parquet. Dataset IDs feed straight into AutoScientist, closing an intent-to-trained-model loop. Check out the Technical details here. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Adaption Labs Introduces ‘Invent a Dataset’: Training Data Generated From a Task Description, Not a Seed Corpus appeared first on MarkTechPost.

Video has been the most expensive modality to reason over. A Gemini model handed a 90-minute lecture has, until now, ingested the whole thing at a fixed one frame per second, whether the question was ‘summarize this’ or ‘what time does the speaker switch to the pricing slide?’ That single-pass design forces a bad trade: pay for the full timeline in context, or pre-chunk the video and risk dropping the detail that mattered. This week, Google launched agentic video understanding across its Flash models. Instead of ingesting the timeline, Gemini navigates it deciding what to watch, at what frame rate, and through which modality. Google reports up to 88% fewer tokens, up to 66% lower cost, and up to 7% higher accuracy on standard video benchmarks. Is it deployable? Yes, but only as a hosted API feature. There are no open weights and nothing to self-host. It ships through the Gemini API in Google AI Studio and the Gemini Enterprise Agent Platform, works with both file uploads and public YouTube URLs, and bills at standard Gemini API token pricing with no additional feature fee. What actually changed Static processing, still the default on every Gemini model, extracts frames at 1 FPS in a single pass, processes audio at 1 Kbps single channel, and inserts timestamps every second. Agentic processing replaces that with a loop. The model pairs its own reasoning with native video tools to search, scan, and inspect target segments across frames, audio, and transcripts, loading only what the prompt requires. Developers could already assemble this by hand; the change is that Gemini runs the loop internally, which is where the development overhead disappears. Across Google’s evaluations, Gemini 3.7 Flash with agentic understanding lands on the accuracy-to-cost Pareto frontier for video analysis among the models tested. The efficiency gains concentrate on long-form content, from 10-minute how-to guides to multi-hour recordings. What the API returns Agentic processing adds two step types to the response steps array: a processing_call when the model requests a segment or transcript, and a matching processing_result when that load completes. They interleave with thought steps and precede model_output, so they can drive a live progress trace in your UI. Their presence is also how you verify agentic mode actually ran. Token accounting splits accordingly. Navigation reasoning bills as thought tokens (total_thought_tokens); frames, audio, and transcripts loaded on demand bill as tool-use tokens (total_tool_use_tokens). Enabling it is one field on the video part: Copy CodeCopiedUse a different Browser interaction = client.interactions.create( model=”gemini-3.7-flash”, input=[ { “type”: “video”, “uri”: “https://youtu.be/7Z5Vy9JBANs”, “processing”: “agentic” }, { “type”: “text”, “text”: “What are the 3 most important announcements in this keynote?”, }, ], ) You can also mix modes per video inside a single request, agentic on the long lecture, static on the short clip. Key Takeaways Agentic mode lets Gemini navigate a video timeline instead of ingesting it at a fixed 1 FPS. Google reports up to 88% fewer tokens, 66% lower cost, and 7% higher accuracy on video benchmarks. Supported on Gemini 3.8, 3.7, 3.6 Flash and 3.5 Flash-Lite; enabled by one processing field. Static remains better for clips under five minutes and for frame-by-frame precision work. Standard API pricing applies, but navigation reasoning is billed as thought tokens. Check out the Google blog, Gemini API video understanding docs, Developer guide in AI Studio and Agentic vision announcement. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Google Launches Agentic Video Understanding for Gemini Flash Models, Cutting Video Tokens by Up to 88% appeared first on MarkTechPost.

GitHub has released Project HydraFusion, a research preview that stops treating model choice as a one-time setting. Instead of routing your prompt to a single model, HydraFusion builds an execution plan per request. It can draft with one model, have a second model critique the draft, or escalate to a stronger model when a quality gate rejects the first attempt. Models come from multiple providers. The developer picks HydraFusion once, the same way they would pick any other model. Is it deployable? Yes, but narrowly. HydraFusion is live as a research preview for users on all GitHub Copilot plans, inside GitHub Copilot CLI only. There are no open weights and no self-hosted path. Run /update, then /experimental on, then /model and select HydraFusion (Research Preview). Billing is per token consumed by whichever models the workflow invokes, at each model’s standard rate. What the system actually does HydraFusion follows Auto model selection, which GitHub shipped earlier in 2026 to match a task to one best-suited model. HydraFusion goes a step further and treats workflow selection as an optimization problem. It reads capability signals for reasoning, code generation, debugging, and tool use. It then picks the least complex workflow expected to clear the quality bar, spending extra model calls only where they are likely to help. The three execution patterns For each request, HydraFusion currently selects one of three patterns: Single: One selected model solves the task directly. Cascade: An efficient model drafts a solution. A quality gate then either accepts it or escalates to a stronger model. Critique: One model drafts, an independent read-only critic from a different model family reviews it, and the drafting model revises once. The review follows the same pattern as Rubber Duck. Each pattern trades quality against cost differently. Single preserves speed. Cascade keeps a path to stronger inference open. Critique adds an outside perspective where review beats another unaided attempt. Engineering guardrails GitHub built the runtime around five operating principles that matter for repository-level work: Complete accounting across every leg, including drafting, critique, revision, escalation, retry, and fallback. Bounded execution with explicit timeout and cancellation per leg. Isolated review, where critics run in tool-less contexts and cannot modify the repository. Fail-safe application, applying no patch when a workflow is cancelled or fails validation. Validated routing, verifying model bindings, fallback behavior, and availability before execution starts. Internally the runtime logs role, outcome, cost, latency, and diagnostics per leg. Externally the developer sees one coherent response and one permission-aware change set. Benchmark results GitHub team evaluated fixed HydraFusion policies on three agentic coding benchmarks, using Claude Opus 5 and GPT-5.6 Sol as baselines. All models ran at medium reasoning level. The reported figures below are relative to Opus 5. Benchmark Estimated cost vs Opus 5 Verified task quality vs Opus 5 TerminalBench 2.1 67% lower +4.9 points DeepSWE 36% lower −1.5 points CheckpointBench 65% lower −0.1 points CheckpointBench is GitHub’s internal multi-turn set, curated from real Copilot sessions and anchored to immutable public commits so runs are replayable. Key Takeaways HydraFusion picks a workflow per request, not just a model, across multiple providers. Three patterns ship today: Single, Cascade with a quality gate, and Critique with a cross-family reviewer. Best result: +4.9 quality points at 67% lower estimated cost on TerminalBench 2.1. On DeepSWE and CheckpointBench it trails Opus 5 slightly while cutting cost 36% and 65%. Available now in Copilot CLI via /experimental, billed at each underlying model’s standard rate. Check out the GitHub Blog announcement, and GitHub Community discussion #206492. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post GitHub Introduces Project HydraFusion: Runtime Multi-Model Orchestration That Builds a Workflow Per Coding Task in Copilot CLI appeared first on MarkTechPost.

arXiv:2606.06350v2 Announce Type: replace Abstract: Reliable rubric grading requires more than accurate score prediction. Each judgement must be grounded in the mark scheme and evidence from the student answer. Existing credit-assignment and intervention methods, primarily designed for self-contained reasoning tasks such as mathematics reasoning, struggle in this setting because they do not identify where grading reasoning goes wrong or how the model’s belief about the final mark changes during reasoning. We propose Evidence-Diagnosed Intervention Training (EDIT), a two-phase framework for training more rubric-faithful LLM graders. First, EDIT-SFT locates problematic reasoning steps using internal model signals: posterior belief over the final mark and input-grounding scores. It then revises only these local steps with help from a rubric checklist. Second, EDIT-RL calibrates the grader with belief-guided reward shaping, penalising large harmful belief drifts while still allowing helpful exploration. Experiments on two real-world, multi-subject grading benchmarks demonstrate that EDIT consistently outperforms strong supervised fine-tuning and reinforcement learning baselines on both in-domain and out-of-domain splits, with ablation studies confirming that internal-state diagnostics drive these gains. Under deterministic rubric-edit interventions, EDIT-SFT is the most rule-responsive of all evaluated systems, and EDIT-RL largely retains this responsiveness while improving accuracy.