Committee

Google Research has released TimesFM-3, a 330 million parameter time series foundation model that forecasts multiple related series in a single forward pass. Every TimesFM checkpoint through 2.5 was univariate: one series, its own history, nothing else. TimesFM-3 is pretrained natively for multivariate forecasting on more than 1 trillion time points, and accepts multiple targets, past covariates, and past-future covariates with no task-specific fine-tuning. It takes the top average rank among pretrained foundation models on GIFT-Eval, fev-bench, and the TIME leaderboard, on both point and probabilistic metrics. https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/ Is it deployable? Partial, the TimesFM repository code is Apache-2.0, but the TimesFM 3.0 weights ship under timesfm-non-commercial-license-v1.0. They are restricted to non-commercial, non-production use. You can benchmark it today. You cannot ship it behind a production forecast API. What changed Every TimesFM release through 2.5 was univariate. It forecast one series from its own history. Most real forecasting problems are not shaped that way. Google’s example is ice cream sales, where related product sales, foot traffic, weather, promotions, and holidays all move the target. TimesFM-3 is pretrained natively for multivariate forecasting. It carries 330 million parameters and was pretrained on more than 1 trillion time points of real and synthetic series. Three input types work zero-shot, with no task-specific fine-tuning: Multiple targets forecast jointly, with point and quantile outputs for each. Past covariates, known only historically, such as past foot traffic. Past-future covariates, whose future values are known, such as a promotion calendar. Architecture: patches, then two kinds of attention The backbone stays a decoder-only transformer. Contiguous points are grouped into patches of 32 steps, then normalized per series so that wildly different scales do not dominate. Target and past-covariate tokens come from a single patch. Past-future covariate tokens use a lookahead trick: the current patch is concatenated with future patches, so the model sees scheduled events before they occur. Tokens then enter a 2D grid and pass through two alternating attention mechanisms: Causal temporal attention runs horizontally. It is strictly causal and confined to earlier tokens inside the same series, which blocks leakage. Full variate attention runs vertically. At a given time step, a token reads every other series at that step, learning cross-series correlations. One forward pass instead of many Earlier TimesFM versions decoded one patch at a time. That adds latency, compute cost, and compounding error. TimesFM-3 uses Contiguous Patch Masking, the training-time masking strategy introduced with TiRex. Masked placeholder tokens are appended for the whole horizon. Targets and past covariates are masked there. Past-future covariates stay visible, so known future signals still reach the model. The alternating attention layers fill every masked horizon patch simultaneously. Each target receives 9 quantiles, the 10th through 90th percentile, at every horizon step. Benchmarks Google evaluated on GIFT-Eval, fev-bench, and the TIME leaderboard, against Chronos-2, the Toto 2.0 family, and TimesFM-2.5. Among pretrained foundation models, TimesFM-3 takes the top average rank on all three, for both point and probabilistic metrics. The package release notes record rank #1 overall on fev-bench across 100 real-world tasks, rank #1 overall on TIME across 50 domain datasets and 98 evaluation tasks, and rank #1 among foundation models on GIFT-Eval. Interactive explainer Key Takeaways TimesFM-3 is a 330M parameter, natively multivariate time series foundation model, pretrained on 1T+ time points. Alternating causal temporal and full variate attention lets it model cross-series dependencies zero-shot. Contiguous Patch Masking produces the entire horizon in one forward pass, with 9 quantiles per step. It ranks #1 among foundation models on GIFT-Eval, fev-bench, and TIME, in univariate and multivariate modes. The weights are non-commercial and non-production only; TimesFM 2.5 remains the Apache-2.0 option for shipping. Check out the Technical details, Model weights on Hugging Face, and the GitHub repo. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Google AI Releases TimesFM-3: A 330M Parameter Zero-Shot Foundation Model For Multivariate Time Series Forecasting appeared first on MarkTechPost.

Anthropic has opened a research preview of the Model Hardware Standard (MHS), a shared specification that lets AI agents discover and operate physical devices. The problem it targets is plumbing. A lab bench or factory cell is assembled from vendors that never planned to interoperate, so every instrument ships its own interface and specialists hand-write bespoke translators between each pair. According to Anthropic team, the setup normally takes weeks to months, and that MHS cuts it to hours or minutes. The Integration Tax Each instrument ships its own programming interface, so specialists hand-write bespoke translators between every pair. Even once wired together, there is no common way for devices to hand state to an agent or be operated safely by one. Anthropic says setup typically runs weeks to months; MHS reduces it to hours or minutes. How it works MHS standardizes the driver — the layer between an OS and a device. It exposes a small primitive set: read (get temperature), write (set temperature), plus discovery, so devices and agents find each other across a network without a translator in between. It also carries knowledge code alone does not encode — the weight of a robot arm, for instance. Driver tags let a user write that in natural language, or have an agent interview them about the setup. The driver compiles those tags into a reference file: what a device measures, what can be adjusted, which safety limits are enforced. Control runs through three mechanisms: the Model Context Protocol, a CLI, and code files. MHS is model-agnostic — any agent harness can reach it via standard protocols. Interactive explainer · Figures verified against Anthropic’s MHS announcement · © Marktechpost What partners measured Genentech automated the BCA protein assay across a liquid handler, robotic arm, and plate reader. Claude ran trial transfers of dyed liquid, read absorbance, scored itself against an expert’s plate using RMSE, and converged on ~140 µL/s for water (0.016 RMSE) and 10 µL/s for viscous BSA (0.181 RMSE) — parameters its automation experts confirmed as reasonable. QuEra Computing is the sharpest number. A bespoke laser-relock script, built over months by a four-person team, worked about 58% of the time at ~150 seconds per attempt. Handed the same problem through MHS, a four-role agent loop ran unattended overnight and produced a deterministic Python script that recovered the lock 695 times out of 700 — 99.3%, hardest cases in 10–14 seconds against 5–10 minutes for a human. Claude also cut the servo’s residual error from a specialist’s 15.7 mV to 1.55 mV; over a 19-hour run its tune never lost lock, while the expert tune unlocked about 1.6 times an hour (QuEra blog). Carnegie Mellon ran dose-response experiments roughly three times faster, orchestrating a liquid handler, plate reader, robotic arm, and cameras across three computers with incompatible interfaces — one with no programmatic interface at all. Driver-writing through to a completed curve, including an autonomous rerun after the agent rejected an R² < 0.9 fit, took about eight hours against the several weeks a vendor setup takes. Six induced fault conditions were all blocked before any device moved. At the University of Washington, a PhD student in the Baker and Pinglay labs connected six instruments in under a week, driver-writing included. Tetsuwan Scientific paired MHS with its ResearchOS platform for qPCR pollution profiling. At Janelia, one microscopy rig went from seven programs launched in a fixed order to a single dashboard click. Key Takeaways A shared driver spec letting AI agents discover and operate any device with a programmable interface. Integration drops from weeks to hours: CMU hit a finished dose-response curve in eight. QuEra’s laser relock: 58% at ~150s, to 99.3% across 700 trials, as a deterministic script. Model-agnostic and MCP-compatible; safety limits live in the driver, not the prompt. Still gated, and still needs supervision — Claude’s physical reasoning has real gaps. Check out the full announcement and apply for the preview. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Anthropic Opens a Research Preview of the Model Hardware Standard (MHS): A Shared Specification for AI Agents to Safely Operate Physical Devices appeared first on MarkTechPost.

MirroS released Code-as-World: a paradigm that represents physical worlds through executable world representations. The argument is narrow and testable: pixels are evidence of a physical scene, not its ontology. A video model can predict plausible frames without ever representing mass, contact, or gravity. So instead of pixels, latents, or captions, Code-as-World represents a scene as executable code — a scene.json that MuJoCo can run, that an agent can verify against the source video, and that anyone can edit and re-simulate. An agentic loop recovers those programs from real footage in up to five rounds. The verified worlds then become training data with exact physical labels, which real video does not carry. Trained on that supervision, Code-as-World-VL-9B scores 55.4 MRA on QuantiPhy-validation, above Gemini-3.1 Flash at 54.8 and roughly 15 points above the strongest open-weight baseline. Is it deployable? Yes, at the research and internal-prototype tier. MirroS shipped the GitHub repo and two checkpoints — Code-as-World-VL-4B and Code-as-World-VL-9B — under Apache 2.0, fine-tuned from Qwen3.5-4B and Qwen3.5-9B. Both are BF16 safetensors served by vLLM behind an OpenAI-compatible /v1 endpoint, with 16 sampled frames per video and –max-model-len 4608. The idea: pixels are evidence, not ontology The MirroS technical report argues that video models, 3D reconstruction, and captions each recover part of a scene but none recovers its mechanism. Code-as-World represents a scene as an executable world representation (EWR), a triple p = (C, E, A): Composition: objects, geometry, metric dimensions, mass, friction, gravity. Floors and walls are static physical entities so they can support and collide. Evolution: initial states, forces, contacts, collisions, termination conditions, duration. Executing it expands composition into a full state trajectory. Appearance: camera, lighting, materials, background, frame rate, render config. Changing it never changes the physics. In the released implementation, that triple compiles into a scene.json executed in MuJoCo, with two interchangeable engines: an animation engine (kinematic poses) and a physics engine (forces and contacts). Agentic discovery instead of one-shot prediction Recovering an EWR from a video is an inverse problem, so the team frames it as abductive search. An agent runs propose → instantiate → execute → render → verify for up to K = 5 rounds. For video input, SAM 3 supplies instance masks and image-plane tracks, VGGT-Omega estimates depth and camera geometry, and SAM 3D generates per-object meshes. Candidate rollouts are projected back into the input view and compared at selected key frames on RGB, depth, masks, and trajectories. Frame-level discrepancies aggregate into structured feedback Δ that guides the next revision; when the budget runs out without acceptance, the hypothesis is rejected. At a matched five-evaluation budget, the loop beats Best-of-5 independent sampling on Visual Alignment, Object IoU, Traj-ADE, and Accuracy@2%D — and the result repeats under the second execution engine. Candidate videos come from WISA-80K after motion-focused filtering; sim-to-real re-rendering uses Wan2.2-VACE plus an internal video model. Verified worlds as training supervision Phase 1 is supervised fine-tuning on 73,335 image-space QA pairs built from RefCOCO/+/g, RefCLEF and GOT-10K, covering extent, position, displacement, velocity and acceleration in raw pixels. Phase 2 applies GRPO to world-space VQA drawn from 1,585 text-driven and 988 video-driven executable worlds, rewarded on scale-normalized numerical accuracy plus unit and format terms. Training used eight NVIDIA H100 GPUs. On QuantiPhy-validation (159 items, MRA macro-averaged over 2S/2D/3S/3D): 4B = 50.6, 9B = 55.4, 27B reasoning = 58.6, against Gemini-3.1 Flash at 54.8, ChatGPT-5.1 at 48.4, and the strongest open-weight baseline Qwen3-VL-32B-Instruct at 40.2. The ablation is the more useful number: image-space-only scores 44.2 (4B) and 50.9 (9B); adding both world-space sources lifts them to 50.6 and 55.4. Pixel-level grounding improves too — the 9B goes 63.7 → 68.3 on RefCOCO and 20.1 → 26.6 on GOT-10K after world-space RL. Key Takeaways Code-as-World turns a video into an editable scene.json that MuJoCo can execute and verify. Five-round propose→verify search beats Best-of-5 sampling at the same compute budget. Verified worlds supply exact physical labels that real video simply does not carry. 9B hits 55.4 MRA on QuantiPhy, above Gemini-3.1 Flash at 54.8; 4B and 9B are Apache 2.0. Rigid-body only, and the model never learns the discovery loop itself. Check out the Technical report, MirroS blog, Project page, GitHub and Announcement on X. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Meet ‘Code-as-World’: An Agentic Loop That Rewrites Real Videos Into Executable MuJoCo Physics Programs appeared first on MarkTechPost.

Time to first token (TTFT) is the metric teams use to pick an inference API for voice. It is also the metric that misleads them. TTFT marks when generation starts; a text-to-speech model cannot speak until a full clause arrives. Between those two points sits the difference between an agent that feels conversational and one that gets interrupted. This piece benchmarks every layer of the voice stack including LLM, speech-to-text, text-to-speech, and speech-to-speech. Why TTFT Is the Right Entry Point and the Wrong Finish Line A voice agent is a latency budget with a language model inside it. Every stage spends milliseconds the user can hear. Time to first token (TTFT) is the interval between sending an inference request and receiving the first token back. IBM’s definition frames it as the moment a system transitions from idle to visibly active. For chat, TTFT is close to the whole story. For voice, it is one term in a sum. The reason is mechanical. A text-to-speech model cannot synthesize half a word. It needs a complete clause or sentence before it produces audio. LiveKit calls the resulting metric time-to-first-sentence (TTFS), and argues in its Gemma 4 deployment post that TTFS is what users actually feel. That gives you two knobs rather than one. TTFT controls when generation starts. Tokens per second controls how fast the first sentence completes. A provider that wins one and loses the other will not feel fast. The Latency Budget: What One Voice Turn Actually Costs LiveKit’s voice agents overview breaks a turn into STT at roughly 100–200ms, LLM at 300–500ms with streaming, TTS at 100–200ms, and network at 50–150ms over WebRTC. It puts the practical end-to-end target at 700ms to 1.2s. Kwindla Hultman Kramer, co-creator of Pipecat, has advised targeting 800ms median voice-to-voice latency, with a looser 1,500ms acceptable for a proof of concept. His rough arithmetic splits that four ways at roughly 200ms each: transport and media processing, STT plus phrase endpointing, LLM inference, and TTS. Daily’s earlier work on the fastest voice bot supplies the human baseline. Typical human response time in conversation is around 500ms. Pauses beyond 800ms start to feel unnatural. Daily’s February 2026 voice-agent LLM benchmark translates that into an LLM requirement directly. Natural conversation needs voice-to-voice under 1,500ms, which works out to roughly 700ms of TTFT budget for a text-mode LLM inside a transcription-to-LLM-to-voice harness. That 700ms number is the bar to hold every provider against. How to Read a TTFT Benchmark Without Being Misled Before the tables, five methodology facts that change what the numbers mean: 1. Workload shape dominates: Artificial Analysis changed its default workload in March 2026. The site now reports 10k input token prompts rather than 1k. Longer prompts raise both TTFT and output speed. LiveKit argues this is closer to reality for voice, because production agents front-load policy, persona, escalation rules, retrieved data, and tool schemas. 2. Server location is baked in: Artificial Analysis tests from a virtual machine in Google Cloud’s us-central1-a zone. It states plainly that TTFT includes network latency and may advantage or disadvantage providers based on where they serve. 3. Reasoning tokens count: In the Artificial Analysis definition, TTFT for a reasoning model is the first reasoning token, not the first answer token. Those are separate columns. 4. Measure from the receiving side: Daily notes that model providers sometimes quote TTFT internal to their inference stacks. Daily measures from request send to first usable token off the API. 5. Runs are not repeatable: Daily is blunt about this: TTFT varies substantially between benchmark runs, and providers change inference stacks and sometimes weights without changing model names. Layer 1: LLM Time to First Token Figures below are from the Artificial Analysis API providers leaderboard, retrieved August 30, 2026. The “first chunk” column is TTFT. Workload is 10k input tokens, single prompt, median over 72 hours. Lowest measured first-chunk latency Provider Model TTFT Output speed Baseten gpt-oss-120b (high) 0.23s 266 tok/s Baseten gpt-oss-120b (low) 0.24s 271 tok/s DeepInfra Nemotron 3 Ultra 0.28s 371 tok/s Cohere North Mini Code 0.32s 104 tok/s Cohere Command A+ 0.40s 239 tok/s Baseten Inkling Small 0.42s 337 tok/s Modular Gemma 4 31B (NVFP4) 0.44s 243 tok/s Nebius GLM-5.3-Flash 0.46s 206 tok/s Fireworks Nemotron 3.5 Lightning 0.46s 501 tok/s Together AI Kimi K2.7 Code 0.47s 245 tok/s Cerebras gpt-oss-120b (high) 0.49s 1,697 tok/s The throughput trap Silicon vendors optimize for a different metric than voice agents need. Provider Model TTFT Output speed Cerebras gpt-oss-120b (high) 0.49s 1,697 tok/s Celeris Celeris-1 0.62s 1,612 tok/s Cerebras Gemma 4 31B 0.53s 1,351 tok/s Groq gpt-oss-20b (high) 0.82s 957 tok/s SambaNova gpt-oss-120b (high) 0.92s 706 tok/s Groq gpt-oss-120b (low) 0.69s 473 tok/s Inception Mercury 2 3.07s 770 tok/s Mercury 2 is the clearest illustration. It is a diffusion-based language model, and it generates 770 tokens per second. Its first chunk arrives at 3.07s. That is four times the entire LLM budget for a natural conversation. Cerebras and Groq are a different case. Their TTFT is respectable and their throughput is exceptional. For TTFS specifically, that combination is strong, because the sentence completes almost immediately after the first token lands. Frontier and proprietary endpoints Provider Model TTFT Output speed Amazon Bedrock GPT-5.6 Luna (non-reasoning) 0.59s 181 tok/s Amazon Bedrock GPT-5.6 Terra (non-reasoning) 0.72s 103 tok/s OpenAI GPT-5.6 Luna (non-reasoning) 0.74s 113 tok/s Google Gemini 3.7 Flash (low), AI Studio 0.84s 315 tok/s Anthropic Claude 4.5 Haiku (non-reasoning) 0.84s 82 tok/s Amazon Bedrock Nova Micro 0.86s 264 tok/s Google Gemini 3.5 Flash (minimal), AI Studio 0.90s 202 tok/s OpenAI GPT-5.6 Sol (non-reasoning) 1.06s 71 tok/s Note the same model on different hosts. GPT-5.6 Luna non-reasoning measures 0.59s on Amazon Bedrock and 0.74s on OpenAI’s own API. Hosting and routing matter as much as the weights. The vendor-measured outlier LiveKit publishes TTFT figures for its own inference product. Gemma 4 31B on LiveKit Inference measured 192ms, against Gemini 2.5 Flash at 911ms, GPT-5.5 at 966ms, GPT-4.1 at 1,006ms, and the same Gemma 4 31B via OpenRouter

Most robotics launches ask you to trust a demo video. Pollen Robotics, the Bordeaux robotics team at Hugging Face, is instead shipping the training loop. This week it opened pre-orders for Microduck, a 25 cm bipedal robot in which every movement — walking, sitting, kicking, roller-skating, standing back up after a fall — is a neural policy trained in a physics simulator and exported to the hardware. It costs $399. The training environments, the reward functions, the domain-randomization settings, and the sim-to-real recipe are all public on GitHub. Microduck follows Reachy Mini, which has shipped more than 10,000 units, but reverses its premise: where Reachy Mini was built to sit on a desk and interact, Microduck is built to leave the desk, fall over, and get back up. The Hardware Microduck is 25 cm tall, 14 cm wide, and under 800 g. It carries 15 motors across legs, neck, and head, plus an articulated beak that picks objects off the floor. Compute is a Rockchip RK3566 with an AI accelerator, 1 GB of RAM, and 32 GB of storage. The sensor stack is unusually complete for the price. A front camera sits behind a dedicated camera-use indicator. Two IMUs are fitted, one in the body and one in the head. Range sensing is a compact LiDAR, an 8×8 time-of-flight matrix. There are microphones and a speaker, two NFC antennas, plus Wi-Fi and Bluetooth. Power is a removable NP-F550 battery, 2600 mAh, good for about an hour. Seven trained moves ship in the box, driven by a bundled game controller before you write code: walk, sit and stand, kick, grab, roller-skate, and self-recovery. The robot does not speak. Each unit generates its own audio identity on first wake and keeps that voice permanently. How the behaviors are actually trained Policies are trained in microduck_rl, built on mjlab (MuJoCo Warp) with PPO. Pollen reports roughly one to two hours on a CUDA GPU for a usable gait at 4096 parallel environments. Without a local GPU, appending –hf-jobs runs the same command on Hugging Face Jobs. The sim-to-real work sits in the actuator model. Each servo uses the BAM M6 model of the Dynamixel XL330 — voltage control law, back-EMF, and Coulomb, Stribeck, and load-dependent friction — rather than an ideal PD controller. Per-environment randomization covers battery voltage, voltage sag under load, command delay, and friction magnitude. Backlash variants train against ±1° of gear play, 2° total, in series with each of the 14 servo joints in the RL layout. Because the real encoder sits on the output side of that play, the observations read through it. Trained policies export to ONNX with the observation normalizer baked into the graph. Pollen warns against deploying hand-converted checkpoints for exactly this reason. On the robot, a Rust runtime drives the 50 Hz control loop and the motor bus. Every policy shares a 61-dimensional actor observation: 48 proprioception dimensions plus commands for twist (3), head pose (4), and body pose (6). That shared contract is what lets walk, recover, and trick policies hot-swap mid-run. Environments that ignore a command slot zero-pad it rather than dropping it. The published registry covers 13 tasks: velocity tracking, stand-up, sit-stand, ground pick, ball kick (70 mm, 15 g ball, actor ball-blind), roulade, and five roller-skating environments. Key Takeaways $399 open-source-software biped, pre-orders open August 27, 2026, deliveries targeted before Christmas. 15 motors, camera, LiDAR, two IMUs, NFC, Wi-Fi/Bluetooth, RK3566, ~1 hour runtime. Policies train in mjlab/MuJoCo Warp with PPO, ~1–2 hours for a gait at 4096 envs. Sim-to-real hinges on a BAM actuator model plus voltage, delay, friction, and ±1° backlash randomization. Software is Apache-2.0; the mechanical and electronic design files are not open. Check out the Microduck product page, launch blog post, press kit and spec sheet, microduck runtime repo, microduck_rl training repo and announcement from Thomas Wolf. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Hugging Face Unveils Microduck: A $399 Open-Source 25 cm Biped You Train with Reinforcement Learning appeared first on MarkTechPost.

Google has released Gemini Omni 1.1 Flash (gemini-omni-1.1-flash), a production update to its native multimodal video generation and editing model. The release moves Omni from a capable generator to a directable one: scene extension now reads up to 10 seconds of prior context instead of a single final frame, first and last frames can be pinned to control camera movement, drafts render in 360p at a third of 720p cost, finals upscale to 4K, and video clips can be passed as references for character consistency. Gemini Omni Flash is built on three properties Google distinguishes from prior video models: native multimodality (text, image, audio, and video processed together), conversational editing through the Interactions API, and world knowledge inherited from Gemini. Editing is stateful — you pass previous_interaction_id and the model applies your change while preserving what you did not mention, without re-uploading the prior video. Is it deployable? It is available through the Gemini API in Google AI Studio and the Gemini Enterprise Agent Platform, with Adobe, Figma Weave, GMI Cloud, and Runway already named as production users. Scene Extension is the Main Change Omni 1.1 analyzes up to 10 seconds of prior context when continuing a clip. Google states that previous models referenced only the final second. Extensions run in 10-second increments to a cumulative 40 seconds, and the model generates a 3–10 second continuation per call. Some final frames of the input are edited to make the seam continuous. The constraints are specific. Extension appends to the end of a clip only — no prepending, no mid-clip insertion. Uploaded input videos must be 10 seconds or shorter, unless you are extending a model-generated video in multi-turn. You cannot add new dialogue when extending an uploaded video where someone is speaking; spoken dialogue is supported in multi-turn extension via previous_interaction_id. Keyframes and Video References You can now supply a first and last frame and have the model generate the continuous video between them, which is the mechanism behind orbits, dolly-zooms, and seamless loops. Prompts bind media to roles with tags: <FIRST_FRAME>, <LAST_FRAME>, <IMAGE_REF_N>, and <VIDEO_REF_N>. Video references accept a maximum of three clips, up to three seconds each, and work best for likenesses. Audio inside a video reference is ignored. Reasoning across multiple videos is not supported and may degrade output. Cost Control: Draft in 360p, Ship in 4K The resolution parameter in response_format takes 360p, 720p (default), 1080p, and 4k, with the top two upscaled. Google reports 360p previews generate up to 60% faster and at a third of the cost of 720p, based on system throughput of 360p versus 720p. That makes the draft-then-upscale loop the intended production pattern: iterate cheaply, render once. Pricing, provenance, and limits Input is $1.50 per 1M tokens (text, image, video, audio). Output is $9.00 per 1M text tokens and $17.50 per 1M video tokens. Video billing runs at 5,792 tokens per second of 720p, an effective ~$0.10 per second under standard pricing. Every generated video carries SynthID watermarking — invisible to viewers, programmatically detectable for provenance. Notable gaps: no system instructions, temperature, top_p, stop sequences, or negative prompts (negatives go in the prompt text); voice editing is unsupported; audio references are unsupported; YouTube URLs cannot be used as a source. English is fully supported; other languages are unevaluated. For outputs above 4MB, use delivery=”uri” and poll the Files API until the file is ACTIVE. Google names Adobe (Firefly), Figma Weave, GMI Cloud, and Runway as customers already running Omni Flash in production. The model is also live in Google Flow for AI Plus, Pro, and Ultra subscribers, with scene extension in the Gemini app. Comparison Key Takeaways Scene extension now reads 10s of prior context, up from one final second, and stacks to 40s total. First/last frame interpolation plus <VIDEO_REF_N> tags give shot-level camera and character control. 360p drafts run up to 60% faster at a third of 720p cost; 1080p and 4K are upscaled outputs. Paid tier only — ~$0.10 per second of 720p video, no free tier, no provisioned throughput. Check out the Google blog announcement, Gemini API Omni documentation, Gemini API pricing, and Omni quickstart cookbook. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Google AI Releases Gemini Omni 1.1 Flash: 40-Second Scene Extension, First/Last Frame Control, and 4K Upscaling appeared first on MarkTechPost.

In this tutorial, we build an ensemble weather forecasting workflow with NVIDIA Earth2Studio. We install the required Earth2Studio components while preserving Colab’s existing CUDA-enabled PyTorch environment, load the FCN prognostic model, and retrieve atmospheric initial conditions from GFS. We then implement a custom wind-power diagnostic that converts 10-meter wind components into turbine capacity factors, along with a variable-scaled perturbation system that applies physically appropriate noise amplitudes to different atmospheric variables while retaining an unperturbed control member. Using Earth2Studio’s low-level iterator, coordinate-mapping, batching, and Zarr APIs, we construct our own ensemble execution pipeline, write forecast and diagnostic fields to a coordinate-aware data store, and verify the forecasts against GFS analyses using latitude-weighted RMSE, fair CRPS, ensemble spread, and spread-skill ratios. Finally, we visualize ensemble uncertainty through spatial maps, geopotential-height spaghetti contours, point-based fan charts, wind-capacity-factor forecasts, and lead-time skill curves. Copy CodeCopiedUse a different Browser import importlib.util, os, subprocess, sys if importlib.util.find_spec(“earth2studio”) is None: import numpy as _np, torch as _torch cfile = os.path.join(os.getcwd(), “e2s_constraints.txt”) with open(cfile, “w”) as f: f.write(f”torch=={_torch.__version__.split(‘+’)[0]}n”) f.write(f”numpy=={_np.__version__}n”) env = {**os.environ, “PIP_CONSTRAINT”: cfile} subprocess.check_call( [sys.executable, “-m”, “pip”, “install”, “-q”, “earth2studio[fcn,data,perturbation,statistics]”], env=env) print(“n>>> Install done. If the imports below fail: Runtime > Restart session, re-run.n”) os.environ.setdefault(“EARTH2STUDIO_CACHE”, “/content/e2s_cache”) os.makedirs(“outputs”, exist_ok=True) from collections import OrderedDict from datetime import datetime, timedelta, timezone from tqdm.auto import tqdm from earth2studio.data import GFS, fetch_data from earth2studio.io import ZarrBackend from earth2studio.models.batch import batch_coords, batch_func from earth2studio.models.px import FCN from earth2studio.statistics import rmse from earth2studio.utils import handshake_coords, handshake_dim from earth2studio.utils.coords import map_coords from earth2studio.utils.time import to_time_array from earth2studio.utils.type import CoordSystem if DEVICE.type == “cpu”: print(“!! No GPU detected — this will be very slow. Runtime > Change runtime type > T4 GPU”) NENSEMBLE = 8 BATCH_SIZE = 2 NSTEPS = 8 SAVE_VARS = [“t2m”, “z500”, “u10m”, “v10m”, “tcwv”] VERIFY_VARS = [“t2m”, “z500”, “u10m”] INIT = (datetime.now(timezone.utc) – timedelta(days=7)).replace() INIT_STR = INIT.strftime(“%Y-%m-%dT%H:%M:%S”) POI = (“New Delhi”, 28.61, 77.21) print(f”Initialization: {INIT_STR} | device: {DEVICE}”) We install Earth2Studio while preserving Colab’s existing CUDA-enabled PyTorch and NumPy environment through package constraints. We configure the model cache, import the forecasting, data, statistics, plotting, and coordinate-management utilities, and detect the available compute device. We also define the ensemble size, batch size, forecast duration, saved variables, verification variables, initialization time, and New Delhi point of interest. Copy CodeCopiedUse a different Browser class WindPowerCF(torch.nn.Module): “””Turbine capacity factor [0,1] from 10 m winds via power-law shear + power curve.””” def __init__(self, lat, lon, hub=100.0, alpha=0.143, cut_in=3.0, rated=12.0, cut_out=25.0): super().__init__() self.lat, self.lon = lat, lon self.hub, self.alpha = hub, alpha self.cut_in, self.rated, self.cut_out = cut_in, rated, cut_out def input_coords(self) -> CoordSystem: return OrderedDict({ “batch”: np.empty(0), “variable”: np.array([“u10m”, “v10m”]), “lat”: self.lat, “lon”: self.lon, }) @batch_coords() def output_coords(self, input_coords: CoordSystem) -> CoordSystem: target = self.input_coords() for i, (key, _) in enumerate(target.items()): if key != “batch”: handshake_dim(input_coords, key, i) handshake_coords(input_coords, target, key) oc = OrderedDict({ “batch”: np.empty(0), “variable”: np.array([“wind_cf”]), “lat”: self.lat, “lon”: self.lon, }) oc[“batch”] = input_coords[“batch”] return oc @batch_func() def __call__(self, x: torch.Tensor, coords: CoordSystem): oc = self.output_coords(coords) u, v = x[…, 0:1, :, :], x[…, 1:2, :, :] ws10 = torch.sqrt(u * u + v * v) ws = ws10 * (self.hub / 10.0) ** self.alpha ramp = (ws ** 3 – self.cut_in ** 3) / (self.rated ** 3 – self.cut_in ** 3) cf = torch.zeros_like(ws) cf = torch.where((ws >= self.cut_in) & (ws < self.rated), ramp.clamp(0, 1), cf) cf = torch.where((ws >= self.rated) & (ws <= self.cut_out), torch.ones_like(cf), cf) return cf, oc class VariableScaledNoise: “””Spatially correlated noise with per-variable amplitudes + control member.””” def __init__(self, amplitudes: dict, default: float = 0.0, control_member: bool = True): self.amplitudes, self.default, self.control = amplitudes, default, control_member try: from earth2studio.perturbation import SphericalGaussian self.sampler, self.kind = SphericalGaussian(noise_amplitude=1.0), “SphericalGaussian” except Exception: from earth2studio.perturbation import Brown self.sampler, self.kind = Brown(noise_amplitude=1.0), “Brown” def __call__(self, x: torch.Tensor, coords: CoordSystem): noise, _ = self.sampler(torch.zeros_like(x), coords) vax = list(coords).index(“variable”) amps = torch.tensor([self.amplitudes.get(str(v), self.default) for v in coords[“variable”]], device=x.device, dtype=x.dtype) shape = [1] * x.ndim; shape[vax] = amps.numel() pert = noise * amps.reshape(shape) if self.control and “ensemble” in coords: eax = list(coords).index(“ensemble”) mask = torch.tensor((np.asarray(coords[“ensemble”]) != 0).astype(np.float32), device=x.device, dtype=x.dtype) mshape = [1] * x.ndim; mshape[eax] = mask.numel() pert = pert * mask.reshape(mshape) return x + pert, coords We create a custom diagnostic model that converts 10-meter wind components into hub-height wind speed and turbine capacity factor. We validate coordinate compatibility through Earth2Studio’s handshake utilities and support batched inputs with the provided decorators. We also implement variable-specific spatial perturbations that retain member zero as an unperturbed control forecast. Copy CodeCopiedUse a different Browser def write_vars(io, x, coords, names): “””Write selected channels of a (…, variable, lat, lon) tensor to the IO backend.””” vax = list(coords).index(“variable”) sub = OrderedDict((k, v) for k, v in coords.items() if k != “variable”) for name in names: hit = np.where(np.asarray(coords[“variable”]) == name)[0] if hit.size: io.write(x.select(vax, int(hit[0])).cpu(), sub, name) def run_ensemble(time, nsteps, nensemble, batch_size, prognostic, diagnostic, perturbation, data, io, save_vars, device): time = to_time_array(time) ic = prognostic.input_coords() x0, c0 = fetch_data(source=data, time=time, lead_time=ic[“lead_time”], variable=ic[“variable”], device=device) print(f”Initial condition tensor: {tuple(x0.shape)} dims={list(c0)}”) oc = prognostic.output_coords(ic) dt = oc[“lead_time”] prog_vars = [v for v in save_vars if v in set(map(str, oc[“variable”]))] total = OrderedDict({ “ensemble”: np.arange(nensemble), “time”: time, “lead_time”: np.asarray([dt * i for i in range(nsteps + 1)]).flatten(), “lat”: oc[“lat”], “lon”: oc[“lon”], }) io.add_array(total, prog_vars + [“wind_cf”]) dx_target = OrderedDict((k, v) for k, v in diagnostic.input_coords().items() if k != “batch”) nbatch = int(np.ceil(nensemble / batch_size)) with torch.inference_mode(): for b in tqdm(range(nbatch), desc=”ensemble batches”): lo = b * batch_size n = min(batch_size, nensemble – lo) x = x0.unsqueeze(0).repeat(n, *([1] * x0.ndim)) coords = OrderedDict({“ensemble”: np.arange(lo, lo + n), **c0}) x, coords = perturbation(x, coords) x, coords = map_coords(x, coords, ic) for step, (xs, cs) in enumerate(prognostic.create_iterator(x, coords)): write_vars(io, xs, cs, prog_vars) xw, cw = map_coords(xs, cs, dx_target) xw, cw = diagnostic(xw, cw) write_vars(io, xw, cw, [“wind_cf”]) if step >= nsteps: break torch.cuda.empty_cache() if device.type == “cuda” else None return io model = FCN.load_model(FCN.load_default_package()).to(DEVICE) grid = model.output_coords(model.input_coords()) LAT, LON = grid[“lat”], grid[“lon”] diagnostic = WindPowerCF(LAT, LON).to(DEVICE) pert = VariableScaledNoise( amplitudes={“t2m”: 0.20, “t850”: 0.20, “z500”: 40.0, “z850”: 25.0, “u10m”: 0.25, “v10m”: 0.25,