YouZum

Uncategorized

AI, Committee, ニュース, Uncategorized

Liquid AI Released LFM2.5-350M: A Compact 350M Parameter Model Trained on 28T Tokens with Scaled Reinforcement Learning

In the current landscape of generative AI, the ‘scaling laws’ have generally dictated that more parameters equal more intelligence. However, Liquid AI is challenging this convention with the release of LFM2.5-350M. This model is actually a technical case study in intelligence density with additional pre-training (from 10T to 28T tokens) and large-scale reinforcement learning The significance of LFM2.5-350M lies in its architecture and training efficiency. While the most AI companies has been focused on frontier models, Liquid AI is targeting the ‘edge’—devices with limited memory and compute—by proving that a 350-million parameter model can outperform models more than twice its size on several evaluated benchmarks. https://www.liquid.ai/blog/lfm2-5-350m-no-size-left-behind Architecture: The Hybrid LIV Backbone The core technical differentiator of the LFM2.5-350M is its departure from the pure Transformer architecture. It utilizes a hybrid structure built on Linear Input-Varying Systems (LIVs). Traditional Transformers rely entirely on self-attention mechanisms, which suffer from quadratic scaling issues: as the context window grows, the memory and computational requirements for the Key-Value (KV) cache increase. Liquid AI addresses this by using a hybrid backbone consisting of: 10 Double-Gated LIV Convolution Blocks: These handle the majority of the sequence processing. LIVs function similarly to advanced Recurrent Neural Networks (RNNs) but are designed to be more parallelizable and stable during training. They maintain a constant-state memory, reducing the I/O overhead. 6 Grouped Query Attention (GQA) Blocks: By integrating a small number of attention blocks, the model retains high-precision retrieval and long-range context handling without the full memory overhead of a standard Transformer. This hybrid approach allows the LFM2.5-350M to support a 32k context window (32,768 tokens) while maintaining an extremely lean memory footprint. Performance and Intelligence Density The LFM2.5-350M was pre-trained on 28 trillion tokens with an extremely high training-to-parameter ratio. This ensures that the model’s limited parameter count is utilized to its maximum potential, resulting in high ‘intelligence density.’ Benchmarks and Use Cases The LFM2.5-350M is a specialist model designed for high-speed, agentic tasks rather than general-purpose reasoning. Benchmark Score IFEval (Instruction Following) 76.96 GPQA Diamond 30.64 MMLU-Pro 20.01 The high IFEval score indicates the model is efficient at following complex, structured instructions, making it suitable for tool use, function calling, and structured data extraction (e.g., JSON). However, the documentation explicitly states that LFM2.5-350M is not recommended for mathematics, complex coding, or creative writing. For those tasks, the reasoning capabilities of larger parameter counts remain necessary. https://www.liquid.ai/blog/lfm2-5-350m-no-size-left-behind Hardware Optimization and Inference Efficiency A major hurdle for AI devs is the ‘memory wall’—the bottleneck created by moving data between the processor and memory. Because the LFM2.5-350M utilizes LIVs and GQA, it drastically reduces KV cache size, boosting throughput. On a single NVIDIA H100 GPU, the model can reach a throughput of 40.4K output tokens per second at high concurrency. Liquid AI team reports device-specific low-memory inference results that make local deployment viable: Snapdragon 8 Elite NPU: 169MB peak memory using RunAnywhere Q4. Snapdragon GPU: 81MB peak memory using RunAnywhere Q4. Raspberry Pi 5: 300MB using Cactus Engine int8. Key Takeaways Extreme Intelligence Density: By training a 350M parameter model on 28 trillion tokens, Liquid AI team achieved an super high 80,000:1 token-to-parameter ratio, allowing it to outperform models more than twice its size on several benchmarks. Hybrid LIV Architecture: The model departs from pure Transformers by using Linear Input-Varying Systems (LIVs) combined with a small number of Grouped Query Attention (GQA) blocks, significantly reducing the memory overhead of the KV cache. Edge-First Efficiency: It is designed for local deployment with a 32k context window and a remarkably low memory footprint—reaching as low as 81MB on mobile GPUs and 169MB on NPUs via specialized inference engines. Specialized Agentic Capability: The model is highly optimized for instruction following (IFEval: 76.96) and tool use, though it is explicitly not recommended for complex coding, mathematics, or creative writing. Massive Throughput: The architectural efficiency enables high-speed utility, processing up to 40.4K output tokens per second on a single H100, making it ideal for high-volume data extraction and real-time classification. Check out the Technical details and Model Weight. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Liquid AI Released LFM2.5-350M: A Compact 350M Parameter Model Trained on 28T Tokens with Scaled Reinforcement Learning appeared first on MarkTechPost.

Liquid AI Released LFM2.5-350M: A Compact 350M Parameter Model Trained on 28T Tokens with Scaled Reinforcement Learning 投稿を読む »

AI, Committee, ニュース, Uncategorized

The gig workers who are training humanoid robots at home

When Zeus, a medical student living in a hilltop city in central Nigeria, returns to his studio apartment from a long day at the hospital, he turns on his ring light, straps his iPhone to his forehead, and starts recording himself. He raises his hands in front of him like a sleepwalker and puts a sheet on his bed. He moves slowly and carefully to make sure his hands stay within the camera frame.  Zeus is a data recorder for Micro1, a US company based in Palo Alto, California that collects real-world data to sell to robotics companies. As companies like Tesla, Figure AI, and Agility Robotics race to build humanoids—robots designed to resemble and move like humans in factories and homes—videos recorded by gig workers like Zeus are becoming the hottest new way to train them.  Micro1 has hired thousands of contract workers in more than 50 countries, including India, Nigeria, and Argentina, where swathes of tech-savvy young people are looking for jobs. They’re mounting iPhones on their heads and recording themselves folding laundry, washing dishes, and cooking. The job pays well by local standards and is boosting local economies, but it raises thorny questions around privacy and informed consent. And the work can be challenging at times—and weird. Zeus found the job in November, when people started talking about it everywhere on LinkedIn and YouTube. “This would be a real nice opportunity to set a mark and give data that will be used to train robots in the future,” he thought.  Zeus is paid $15 an hour, which is good income in Nigeria’s strained economy with high unemployment rates. But as a bright-eyed student dreaming of becoming a doctor, he finds ironing his clothes for hours every day boring.  “I really [do] not like it so much,” he says. “I’m the kind of person that requires … a technical job that requires me to think.”  Zeus, and all the workers interviewed by MIT Technology Review, asked to be referred to only by pseudonyms because they were not authorized to talk about their work. Humanoid robots are notoriously hard to build because manipulating physical objects is a difficult skill to master. But the rise of large language models underlying chatbots like ChatGPT has inspired a paradigm shift in robotics. Just as large language models learned to generate words by being trained on vast troves of text scraped from the internet, many researchers believe that humanoid robots can learn to interact with the world by being trained on massive amounts of movement data.  Editor’s note: In a recent poll, MIT Technology Review readers selected humanoid robots as the 11th breakthrough for our 2026 list of 10 Breakthrough Technologies. Robotics requires far more complex data about the physical world, though, and that is much harder to find. Virtual simulations can train robots to perform acrobatics, but not how to grasp and move objects, because simulations struggle to model physics with perfect accuracy. For robots to work in factories and serve as housekeepers, real-world data, however time-consuming and expensive to collect, may be what we need.  Investors are pouring money feverishly into solving this challenge, spending over $6 billion on humanoid robots in 2025. And at-home data recording is becoming a booming gig economy around the world. Data companies like Scale AI and Encord are recruiting their own armies of data recorders, while DoorDash pays delivery drivers to film themselves doing chores. And in China, workers in dozens of state-owned robot training centers wear virtual-reality headsets and exoskeletons to teach humanoid robots how to open a microwave and wipe down the table.  “There is a lot of demand, and it’s increasing really fast,” says Ali Ansari, CEO of Micro1. He estimates that robotics companies are now spending more than $100 million each year to buy real-world data from his company and others like it. A day in the life Workers at Micro1 are vetted by an AI agent named Zara that conducts interviews and reviews samples of chore videos. Every week, they submit videos of themselves doing chores around their homes, following a list of instructions about things like keeping their hands visible and moving at natural speed. The videos are reviewed by both AI and a human and are either accepted or rejected. They’re then annotated by AI and a team of hundreds of humans who label the actions in the footage. “There is a lot of demand, and it’s increasing really fast.” Ali Ansari, CEO of Micro1  Because this approach to training robots is in its infancy, it’s not clear yet what makes good training data. Still, “you need to give lots and lots of variations for the robot to generalize well for basic navigation and manipulation of the world,” says Ansari. But many workers say that creating a variety of “chore content” in their tiny homes is a challenge. Zeus, a scrappy student living in a humble studio, struggles to record anything beyond ironing his clothes every day. Arjun, a tutor in Delhi, India, takes an hour to make a 15-minute video because he spends so much time brainstorming new chores. “How much content [can be made] in the home? How much content?” he says.  There’s also the sticky question of privacy. Micro1 asks workers not to show their faces to the camera or reveal personal information such as names, phone numbers, and birth dates. Then it uses AI and human reviewers to remove anything that slips through.  But even without faces, the videos capture an intimate slice of workers’ lives: the interiors of their homes, their possessions, their routines. And understanding what kind of personal information they might be recording while they’re busy doing chores on camera can be tricky. Reviews of such footage might not filter out sensitive information beyond the most obvious identifiers. For workers with families, keeping private life off camera is a constant negotiation. Arjun, a father of two daughters, has to wrangle his chaotic two-year-old out of frame. “Sometimes it’s very difficult to

The gig workers who are training humanoid robots at home 投稿を読む »

AI, Committee, ニュース, Uncategorized

Hugging Face Releases TRL v1.0: A Unified Post-Training Stack for SFT, Reward Modeling, DPO, and GRPO Workflows

Hugging Face has officially released TRL (Transformer Reinforcement Learning) v1.0, marking a pivotal transition for the library from a research-oriented repository to a stable, production-ready framework. For AI professionals and developers, this release codifies the Post-Training pipeline—the essential sequence of Supervised Fine-Tuning (SFT), Reward Modeling, and Alignment—into a unified, standardized API. In the early stages of the LLM boom, post-training was often treated as an experimental ‘dark art.’ TRL v1.0 aims to change that by providing a consistent developer experience built on three core pillars: a dedicated Command Line Interface (CLI), a unified Configuration system, and an expanded suite of alignment algorithms including DPO, GRPO, and KTO. The Unified Post-Training Stack Post-training is the phase where a pre-trained base model is refined to follow instructions, adopt a specific tone, or exhibit complex reasoning capabilities. TRL v1.0 organizes this process into distinct, interoperable stages: Supervised Fine-Tuning (SFT): The foundational step where the model is trained on high-quality instruction-following data to adapt its pre-trained knowledge to a conversational format. Reward Modeling: The process of training a separate model to predict human preferences, which acts as a ‘judge’ to score different model responses. Alignment (Reinforcement Learning): The final refinement where the model is optimized to maximize preference scores. This is achieved either through “online” methods that generate text during training or “offline” methods that learn from static preference datasets. Standardizing the Developer Experience: The TRL CLI One of the most significant updates for software engineers is the introduction of a robust TRL CLI. Previously, engineers were required to write extensive boilerplate code and custom training loops for every experiment. TRL v1.0 introduces a config-driven approach that utilizes YAML files or direct command-line arguments to manage the training lifecycle. The trl Command The CLI provides standardized entry points for the primary training stages. For instance, initiating an SFT run can now be executed via a single command: Copy CodeCopiedUse a different Browser trl sft –model_name_or_path meta-llama/Llama-3.1-8B –dataset_name openbmb/UltraInteract –output_dir ./sft_results This interface is integrated with Hugging Face Accelerate, which allows the same command to scale across diverse hardware configurations. Whether running on a single local GPU or a multi-node cluster utilizing Fully Sharded Data Parallel (FSDP) or DeepSpeed, the CLI manages the underlying distribution logic. TRLConfig and TrainingArguments Technical parity with the core transformers library is a cornerstone of this release. Each trainer now features a corresponding configuration class—such as SFTConfig, DPOConfig, or GRPOConfig—which inherits directly from transformers.TrainingArguments. Alignment Algorithms: Choosing the Right Objective TRL v1.0 consolidates several reinforcement learning methods, categorizing them based on their data requirements and computational overhead. Algorithm Type Technical Characteristic PPO Online Requires Policy, Reference, Reward, and Value (Critic) models. Highest VRAM footprint. DPO Offline Learns from preference pairs (chosen vs. rejected) without a separate Reward model. GRPO Online An on-policy method that removes the Value (Critic) model by using group-relative rewards. KTO Offline Learns from binary “thumbs up/down” signals instead of paired preferences. ORPO (Exp.) Experimental A one-step method that merges SFT and alignment using an odds-ratio loss. Efficiency and Performance Scaling To accommodate models with billions of parameters on consumer or mid-tier enterprise hardware, TRL v1.0 integrates several efficiency-focused technologies: PEFT (Parameter-Efficient Fine-Tuning): Native support for LoRA and QLoRA enables fine-tuning by updating a small fraction of the model’s weights, drastically reducing memory requirements. Unsloth Integration: TRL v1.0 leverages specialized kernels from the Unsloth library. For SFT and DPO workflows, this integration can result in a 2x increase in training speed and up to a 70% reduction in memory usage compared to standard implementations. Data Packing: The SFTTrainer supports constant-length packing. This technique concatenates multiple short sequences into a single fixed-length block (e.g., 2048 tokens), ensuring that nearly every token processed contributes to the gradient update and minimizing computation spent on padding. The trl.experimental Namespace Hugging Face team has introduced the trl.experimental namespace to separate production-stable tools from rapidly evolving research. This allows the core library to remain backward-compatible while still hosting cutting-edge developments. Features currently in the experimental track include: ORPO (Odds Ratio Preference Optimization): An emerging method that attempts to skip the SFT phase by applying alignment directly to the base model. Online DPO Trainers: Variants of DPO that incorporate real-time generation. Novel Loss Functions: Experimental objectives that target specific model behaviors, such as reducing verbosity or improving mathematical reasoning. Key Takeaways TRL v1.0 standardizes LLM post-training with a unified CLI, config system, and trainer workflow. The release separates a stable core from experimental methods such as ORPO and KTO. GRPO reduces RL training overhead by removing the separate critic model used in PPO. TRL integrates PEFT, data packing, and Unsloth to improve training efficiency and memory usage. The library makes SFT, reward modeling, and alignment more reproducible for engineering teams. Check out the Technical details.  Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Hugging Face Releases TRL v1.0: A Unified Post-Training Stack for SFT, Reward Modeling, DPO, and GRPO Workflows appeared first on MarkTechPost.

Hugging Face Releases TRL v1.0: A Unified Post-Training Stack for SFT, Reward Modeling, DPO, and GRPO Workflows 投稿を読む »

AI, Committee, ニュース, Uncategorized

Google AI Releases Veo 3.1 Lite: Giving Developers Low Cost High Speed Video Generation via The Gemini API

Google has announced the release of Veo 3.1 Lite, a new model tier within its generative video portfolio designed to address the primary bottleneck for production-scale deployments: pricing. While the generative video space has seen rapid progress in visual fidelity, the cost per second of generated content has remained high, often prohibitive for developers building high-volume applications. Veo 3.1 Lite is now available via the Gemini API and Google AI Studio for users in the paid tier. By offering the same generation speed as the existing Veo 3.1 Fast model at approximately half the cost, Google is positioning this model as the standard for developers focused on programmatic video generation and iterative prototyping. https://blog.google/innovation-and-ai/technology/ai/veo-3-1-lite/ Technical Architecture: The Diffusion Transformer (DiT) The most significant aspect of the Veo 3.1 family is its underlying Diffusion Transformer (DiT) architecture. Traditional generative video models often relied on U-Net-based diffusion, which can struggle with high-dimensional data and long-range temporal dependencies. Veo 3.1 Lite utilizes a transformer-based backbone that operates on spatio-temporal patches. In this architecture, video frames are not processed as static 2D images but as a continuous sequence of tokens in a latent space. By applying self-attention across these patches, the model maintains better temporal consistency. This ensures that objects, lighting, and textures remain coherent across the duration of the clip, reducing the artifacts commonly seen in earlier models. The model performs its computation in a compressed latent space rather than pixel space. This allows the model to handle the high computational demands of video generation while maintaining a lower memory footprint. For developers, this translates to a model that can generate high-definition content without the exponential increase in compute time that usually accompanies resolution scaling. Performance and Output Specifications Veo 3.1 Lite provides specific parameters for resolution and duration, allowing AI devs to integrate it into structured workflows. Unlike the flagship Veo 3.1 model, which supports 4K resolution, the Lite version is optimized for high-definition (HD) outputs. Supported Resolutions: 720p and 1080p. Aspect Ratios: Native support for both landscape (16:9) and portrait (9:16) orientations. Clip Durations: Developers can specify generation lengths of 4, 6, or 8 seconds. Prompt Adherence: The model is optimized for ‘Cinematic Control,’ recognizing technical directives such as ‘pan,’ ’tilt,’ and specific lighting instructions. The ‘Lite’ tag does not refer to a reduction in generation speed compared to the ‘Fast’ tier. Instead, it refers to an optimized parameter set that allows Google team to offer the model at a significantly lower price point while maintaining the same low-latency performance characteristics of Veo 3.1 Fast. The Pricing Shift: Democratizing Video Inference The core value proposition of Veo 3.1 Lite is its cost structure. In the current market, high-quality video inference often costs several dollars per minute of footage, making it difficult to justify for applications like dynamic ad generation or social media automation. Veo 3.1 Lite pricing is structured as follows: 720p: $0.05 per second. 1080p: $0.08 per second. Deployment via Gemini API and AI Studio The accessibility is handled through the Gemini API. This allows for the integration of video generation into existing Python or Node.js applications using standard REST or gRPC calls. One critical technical feature for enterprise developers is the inclusion of SynthID. Developed by Google DeepMind, SynthID is a tool for watermarking and identifying AI-generated content. It embeds a digital watermark directly into the pixels of the video that is imperceptible to the human eye but detectable by specialized software. This is a mandatory component for developers concerned with safety, compliance, and distinguishing synthetic media from captured footage. Key Takeaways Half the Cost, Same Speed: Offers the same low-latency performance as the ‘Fast’ tier at less than 50% of the price ($0.05/sec for 720p). Scalable HD Output: Supports 720p and 1080p resolutions in 4, 6, or 8-second clips with native 16:9 and 9:16 aspect ratios. Architecture: Built on a Diffusion Transformer (DiT) using spatio-temporal patches for superior motion and physical consistency. Developer Ready: Available now via Gemini API (paid tier) and Google AI Studio, featuring built-in SynthID digital watermarking. Check out the Technical details. You can access the model via paid tier on the Gemini API and Google AI Studio. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Google AI Releases Veo 3.1 Lite: Giving Developers Low Cost High Speed Video Generation via The Gemini API appeared first on MarkTechPost.

Google AI Releases Veo 3.1 Lite: Giving Developers Low Cost High Speed Video Generation via The Gemini API 投稿を読む »

AI, Committee, ニュース, Uncategorized

The Download: gig workers training humanoids, and better AI benchmarks

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. The gig workers who are training humanoid robots at home  When Zeus, a medical student in Nigeria, returns to his apartment from a long day at the hospital, he straps his iPhone to his forehead and records himself doing chores.  Zeus is a data recorder for Micro1, which sells the data he collects to robotics firms. As these companies race to build humanoids, videos from workers like Zeus have become the hottest new way to train them.   Micro1 has hired thousands of them in more than 50 countries, including India, Nigeria, and Argentina. The jobs pay well locally, but raise thorny questions around privacy and informed consent. The work can be challenging—and weird. Read the full story.  —Michelle Kim  Our readers recently voted humanoid robots the “11th breakthrough” to add to our 2026 list of 10 Breakthrough Technologies. Check out what else officially made the cut.  AI benchmarks are broken. Here’s what we need instead.  For decades, AI has been evaluated based on whether it can outperform humans on isolated problems. But it’s seldom used this way in the real world.  While AI is assessed in a vacuum, it operates in messy, complex, multi-person environments over time. This misalignment leads us to misunderstand its capabilities, risks, and impacts.  We need new benchmarks that assess AI’s performance over longer horizons within human teams, workflows, and organizations. Here’s a proposal for one such approach: Human–AI, Context-Specific Evaluation.   —Angela Aristidou, professor at University College London and faculty fellow at the Stanford Digital Economy Lab and the Stanford Human-Centered AI Institute.  MIT Technology Review Narrated: can quantum computers now solve health care problems? We’ll soon find out.  In a laboratory on the outskirts of Oxford, a quantum computer built from atoms and light awaits its moment. The device is small but powerful—and also very valuable. Infleqtion, the company that owns it, is hoping its abilities will win $5 million at a competition.   The prize will go to the quantum computer that can solve real health care problems that “classical” computers cannot. But there can be only one big winner—if there is a winner at all.  —Michael Brooks  This is our latest story to be turned into an MIT Technology Review Narrated podcast, which we’re publishing each week on Spotify and Apple Podcasts. Just navigate to MIT Technology Review Narrated on either platform, and follow us to get all our new content as it’s released.  The must-reads  I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology.  1 OpenAI just closed the biggest funding round in Silicon Valley history It raised $122 billion ahead of its blockbuster IPO, which is expected later this year. (WSJ $) + It’s also prepping a push to “rethink the social contract.” (Vanity Fair $) + Campaigners are urging people to quit ChatGPT. (MIT Technology Review)    2 Iran has threatened to attack 18 US tech companies  It’s eyeing their operations in the Middle East. (Politico) + Targets include Nvidia, Apple, Microsoft, and Google. (Engadget) + Iran struck AWS data centers earlier this month. (Reuters $)  3 Artemis II is about to fly humans to the Moon. Here’s the science they’ll do Their experiments will set the stage for future explorers. (Nature) + You can watch the launch attempt today. (Engadget)   4 Putin is trying to take full control of Russia’s internet New outages and blockages are cutting the country off from the world. (NYT $) + Can we repair the internet? (MIT Technology Review)  5 A robotaxi outage in China left passengers stranded on highways  Baidu vehicles froze on the streets of Wuhan. (Bloomberg $) + Police are blaming a “system failure.” (Reuters $)  6 US government requests for social media user data are soaring They’ve skyrocketed by 770% in the past decade. (Bloomberg $) + Is the Pentagon allowed to surveil Americans with AI? (MIT Technology Review)  7 Tesla has admitted that humans sometimes drive its robotaxis Remote drivers occasionally control them completely. (Wired $)  8 A satellite-smashing chain reaction could spiral out of control This data visualization captures the dangers of space collisions. (Guardian) + Here’s all the stuff we’ve put into space. (MIT Technology Review)  9 Meta’s smartglasses can turn you into a creep According to one journalist who wore them for a month. (Guardian)  10 A Claude Code leak has exposed plans for a virtual pet  We could be getting a Tamagotchi for the GenAI era. (The Verge)  Quote of the day  “From now on, for every assassination, an American company will be destroyed.”  —Iran’s Islamic Revolutionary Guard Corps (IRGC) threatens US tech firms in an affiliated Telegram, per CNBC.  One More Thing  ACKERMAN + GRUBER How one mine could unlock billions in EV subsidies  In a pine farm north of the tiny town of Tamarack, Minnesota, Talon Metals has uncovered one of America’s densest nickel deposits. Now it wants to begin mining the ore.  Products made from the nickel could net more than $26 billion in subsidies through the Inflation Reduction Act (IRA), which is starting to transform the US economy. To understand how, we tallied up the potential tax credits available. Read the full story to find out what we discovered.  —James Temple  We can still have nice things  A place for comfort, fun and distraction to brighten up your day. (Got any ideas? Drop me a line.)  + A selfless group of gluttons tried to taste-test every potato chip in the world.  + Get romantic inspiration from these penguins’ engagement pebbles. + Good news: global terrorism has hit a 15-year low. + Enjoy endless new views through these windows around the world.  

The Download: gig workers training humanoids, and better AI benchmarks 投稿を読む »

AI, Committee, ニュース, Uncategorized

Alibaba Qwen Team Releases Qwen3.5 Omni: A Native Multimodal Model for Text, Audio, Video, and Realtime Interaction

The landscape of multimodal large language models (MLLMs) has shifted from experimental ‘wrappers’—where separate vision or audio encoders are stitched onto a text-based backbone—to native, end-to-end ‘omnimodal’ architectures. Alibaba Qwen team latest release, Qwen3.5-Omni, represents a significant milestone in this evolution. Designed as a direct competitor to flagship models like Gemini 3.1 Pro, the Qwen3.5-Omni series introduces a unified framework capable of processing text, images, audio, and video simultaneously within a single computational pipeline. The technical significance of Qwen3.5-Omni lies in its Thinker-Talker architecture and its use of Hybrid-Attention Mixture of Experts (MoE) across all modalities. This approach enables the model to handle massive context windows and real-time interaction without the traditional latency penalties associated with cascaded systems. Model Tiers The series is offered in three sizes to balance performance and cost: Plus: High-complexity reasoning and maximum accuracy. Flash: Optimized for high-throughput and low-latency interaction. Light: A smaller variant for efficiency-focused tasks. https://qwen.ai/blog?id=qwen3.5-omni The Thinker-Talker Architecture: A Unified MoE Framework At the core of Qwen3.5-Omni is a bifurcated yet tightly integrated architecture consisting of two main components: the Thinker and the Talker. In previous iterations, multimodal models often relied on external pre-trained encoders (such as Whisper for audio). Qwen3.5-Omni moves beyond this by utilizing a native Audio Transformer (AuT) encoder. This encoder was pre-trained on more than 100 million hours of audio-visual data, providing the model with a grounded understanding of temporal and acoustic nuances that traditional text-first models lack. Hybrid-Attention Mixture of Experts (MoE) Both the Thinker and the Talker leverage Hybrid-Attention MoE. In a standard MoE setup, only a subset of parameters (the ‘experts’) are activated for any given token, which allows for a high total parameter count with lower active computational costs. By applying this to a hybrid-attention mechanism, Qwen3.5-Omni can effectively weigh the importance of different modalities (e.g., focusing more on visual tokens during a video analysis task) while maintaining the throughput required for streaming services. This architecture supports a 256k long-context input, enabling the model to ingest and reason over: Over 10 hours of continuous audio. Over 400 seconds of 720p audio-visual content (sampled at 1 FPS). Benchmarking Performance: The ‘215 SOTA’ Milestone One of the most highlighted technical claims regarding the flagship Qwen3.5-Omni-Plus model is its performance on the global leaderboard. The model achieved State-of-the-Art (SOTA) results on 215 audio and audio-visual understanding, reasoning, and interaction subtasks. These 215 SOTA wins are not merely a measure of broad evaluation but span specific technical benchmarks, including: 3 audio-visual benchmarks and 5 general audio benchmarks. 8 ASR (Automatic Speech Recognition) benchmarks. 156 language-specific Speech-to-Text Translation (S2TT) tasks. 43 language-specific ASR tasks. According to their official technical reports, Qwen3.5-Omni-Plus surpasses Gemini 3.1 Pro in general audio understanding, reasoning, recognition, and translation. In audio-visual understanding, it achieves parity with Google’s flagship, while maintaining the core text and visual performance of the standard Qwen3.5 series. https://qwen.ai/blog?id=qwen3.5-omni Technical Solutions for Real-Time Interaction Building a model that can ‘talk’ and ‘hear’ in real-time requires solving specific engineering challenges related to streaming stability and conversational flow. ARIA: Adaptive Rate Interleave Alignment A common failure mode in streaming voice interaction is ‘speech instability.’ Because text tokens and speech tokens have different encoding efficiencies, a model may misread numbers or stutter when attempting to synchronize its text reasoning with its audio output. To address this, Alibaba Qwen team developed ARIA (Adaptive Rate Interleave Alignment). This technique dynamically aligns text and speech units during generation. By adjusting the interleave rate based on the density of the information being processed, ARIA improves the naturalness and robustness of speech synthesis without increasing latency. Semantic Interruption and Turn-Taking For AI developers building voice assistants, handling interruptions is notoriously difficult. Qwen3.5-Omni introduces native turn-taking intent recognition. This allows the model to distinguish between ‘backchanneling’ (non-meaningful background noise or listener feedback like ‘uh-huh’) and an actual semantic interruption where the user intends to take the floor. This capability is baked directly into the model’s API, enabling more human-like, full-duplex conversations. Emergent Capability: Audio-Visual Vibe Coding Perhaps the most unique feature identified during the native multimodal scaling of Qwen3.5-Omni is Audio-Visual Vibe Coding. Unlike traditional code generation that relies on text prompts, Qwen3.5-Omni can perform coding tasks based directly on audio-visual instructions. For instance, a developer could record a video of a software UI, verbally describe a bug while pointing at specific elements, and the model can directly generate the fix. This emergence suggests that the model has developed a cross-modal mapping between visual UI hierarchies, verbal intent, and symbolic code logic. Key Takeaways Qwen3.5-Omni uses a native Thinker-Talker multimodal architecture for unified text, audio, and video processing. The model supports 256k context, 10+ hours of audio, and 400+ seconds of 720p video at 1 FPS. Alibaba reports speech recognition in 113 languages/dialects and speech generation in 36 languages/dialects. Key system features include semantic interruption, turn-taking intent recognition, TMRoPE, and ARIA for realtime interaction. Check out the Technical details, Qwenchat, Online demo on HF and Offline demo on HF. Also, feel free to follow us on Twitter and don’t forget to join our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. The post Alibaba Qwen Team Releases Qwen3.5 Omni: A Native Multimodal Model for Text, Audio, Video, and Realtime Interaction appeared first on MarkTechPost.

Alibaba Qwen Team Releases Qwen3.5 Omni: A Native Multimodal Model for Text, Audio, Video, and Realtime Interaction 投稿を読む »

AI, Committee, ニュース, Uncategorized

AI benchmarks are broken. Here’s what we need instead.

For decades, artificial intelligence has been evaluated through the question of whether machines outperform humans. From chess to advanced math, from coding to essay writing, the performance of AI models and applications is tested against that of individual humans completing tasks.  This framing is seductive: An AI vs. human comparison on isolated problems with clear right or wrong answers is easy to standardize, compare, and optimize. It generates rankings and headlines.  But there’s a problem: AI is almost never used in the way it is benchmarked. Although   researchers and industry have started to improve benchmarking by moving beyond static tests to more dynamic evaluation methods, these  innovations resolve only part of the issue. That’s because they still evaluate AI’s performance outside the human teams and organizational workflows where its real-world performance ultimately unfolds.  While AI is evaluated at the task level in a vacuum, it is used in messy, complex environments where it usually interacts with more than one person. Its performance (or lack thereof) emerges only over extended periods of use. This misalignment leaves us misunderstanding AI’s capabilities, overlooking systemic risks, and misjudging its economic and social consequences. To mitigate this, it’s time to shift from narrow methods to benchmarks that assess how AI systems perform over longer time horizons within human teams, workflows, and organizations. I have studied real-world AI deployment since 2022 in small businesses and health, humanitarian, nonprofit, and higher-education organizations in the UK, the United States, and Asia, as well as within leading AI design ecosystems in London and Silicon Valley. I propose a different approach, which I call HAIC benchmarks—Human–AI, Context-Specific Evaluation. What happens when AI fails  For governments and businesses, AI benchmark scores appear more objective than vendor claims. They’re a critical part of determining whether an AI model or application is “good enough” for real-world deployment. Imagine an AI model that achieves impressive technical scores on the most cutting-edge benchmarks—98% accuracy, groundbreaking speed, compelling outputs. On the strength of these results, organizations may decide to adopt the model, committing sizable financial and technical resources to purchasing and integrating it.  But then, once it’s adopted, the gap between benchmark and real-world performance quickly becomes visible. For example, take the swathe of FDA-approved AI models that can read medical scans faster and more accurately than an expert radiologist. In the radiology units of hospitals from the heart of California to the outskirts of London, I witnessed staff using highly ranked radiology AI applications. Repeatedly, it took them extra time to interpret AI’s outputs alongside hospital-specific reporting standards and nation-specific regulatory requirements. What appeared as a productivity-enhancing AI tool when tested in a vacuum introduced delays in practice.  It soon became clear that the benchmark tests on which medical AI models are assessed do not capture how medical decisions are actually made. Hospitals rely on multidisciplinary teams—radiologists, oncologists, physicists, nurses—who jointly review patients. Treatment planning rarely hinges on a static decision; it evolves as new information emerges over days or weeks. Decisions often arise through constructive debate and trade-offs between professional standards, patient preferences, and the shared goal of long-term patient well-being. No wonder even highly scored AI models struggle to deliver the promised performance once they encounter the complex, collaborative processes of real clinical care. The same pattern emerges in my research across other sectors: When embedded within real-world work environments, even AI models that perform brilliantly on standardized tests don’t perform as promised.  When high benchmark scores fail to translate into real-world performance, even the most highly scored AI is soon abandoned to what I call the “AI graveyard.” The costs are significant: Time, effort and money end up being wasted. And over time, repeated experiences like this erode organizational confidence in AI and—in critical settings such as health—may erode broader public trust in the technology as well.  When current benchmarks provide only a partial and potentially misleading signal of an AI model’s readiness for real-world use, this creates regulatory blind spots: Oversight is shaped by metrics that do not reflect reality. It also leaves organizations and governments to shoulder the risks of testing AI in sensitive real-world settings, often with limited resources and support.  How to build better tests  To close the gap between benchmark and real-world performance, we must pay attention to the actual conditions in which AI models will be used. The critical questions: Can AI function as a productive participant within human teams? And can it generate sustained, collective value?  Through my research on AI deployment across multiple sectors, I have seen a number of organizations already moving—deliberately and experimentally—toward the HAIC benchmarks I favor.  HAIC benchmarks reframe current benchmarking in four ways:  1.     From individual and single-task performance to team and workflow performance (shifting the unit of analysis) 2.     From one-off testing with right/wrong answers to long-term impacts (expanding the time horizon) 3.     From correctness and speed to organizational outcomes, coordination quality, and error detectability (expanding outcome measures) 4.     From isolated outputs to upstream and downstream consequences (system effects) Across the organizations where this approach has emerged and started to be applied, the first step is shifting the unit of analysis.  For example, in one UK hospital system in the period 2021–2024, the question expanded from whether a medical AI application improves diagnostic accuracy to how the presence of AI within the hospital’s multidisciplinary teams affects not only accuracy but also coordination and deliberation. The hospital specifically assessed coordination and deliberation in human teams using and not using AI. Multiple stakeholders (within and outside the hospital) decided on metrics like how AI influences collective reasoning, whether it surfaces overlooked considerations, whether it strengthens or weakens coordination, and whether it changes established risk and compliance practices.  This shift is fundamental. It matters a lot in high-stakes contexts where system-level effects matter more than task-level accuracy. It also matters for the economy. It may help recalibrate inflated expectations of sweeping productivity gains that are so far predicated largely on the

AI benchmarks are broken. Here’s what we need instead. 投稿を読む »

AI, Committee, ニュース, Uncategorized

The Download: AI health tools and the Pentagon’s Anthropic culture war

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. There are more AI health tools than ever—but how well do they work?  In the last few months alone, Microsoft, Amazon, and OpenAI have all launched medical chatbots.  There’s a clear demand for these tools, given how hard it is for many people to access advice through the existing medical system—and they could make safe and useful recommendations. But concerns have surfaced about how little external evaluation they undergo before being released to the public.   Read the full story to understand what’s at stake.  —Grace Huckins  The Pentagon’s culture war tactic against Anthropic has backfired  A judge has temporarily blocked the Pentagon from labeling Anthropic a supply chain risk and ordering government agencies to stop using its AI. Her intervention suggests that the feud never needed to reach such a frenzy.  It did so because the government disregarded the existing process for such disputes—and fueled the fire on social media. Find out how it happened and what comes next.  —James O’Donnell  This story is from The Algorithm, our weekly newsletter giving you the inside track on all things AI. Sign up to receive it in your inbox every Monday.  The must-reads  I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology.  1 California has defied Trump to impose new AI regulations Governor Newsom signed off on the new standards yesterday.  (Guardian) + Firms seeking state contracts will need extra safeguards. (Reuters $) + States are installing guardrails despite Trump’s order to stop. (NYT $)  + An AI regulation war is brewing in the US. (MIT Technology Review)   2 Experiments have verified quantum simulations for the first time It’s a breakthrough for quantum computing applications. (Nature) + Which could one day help solve healthcare problems. (MIT Technology Review)  3 The new White House app is a security and privacy nightmare It extensively tracks users and relies on external code. (Gizmodo) + The new app promises “unparalleled access” to Trump. (CNET) + It also invites users to report people to ICE. (The Verge)  4 Big Tech’s $635 billion AI spending faces an energy shock test The Middle East crisis is clouding prospects for growth. (Reuters $) + Here are three big unknowns about AI’s energy burden. (MIT Technology Review)  5 Meta and Google have been accused of breaking child safety rules Australia suspects they flouted a social media ban. (Bloomberg $) + Indonesia is also investigating non-compliance. (Reuters $)  6 Nebius is building a $10 billion AI data center in Finland The company is rapidly expanding Europe’s AI infrastructure. (CNBC)  7 South Korea’s chipmakers’ helium stocks will last until June Beyond that? Who knows. (Reuters $) + Shortages caused by the Iran war threaten the chip industry. (NYT $)   8 Another Starlink satellite has inexplicably exploded  SpaceX suffered a similar episode in December. (The Verge) + We went inside Ukraine’s largest Starlink repair shop. (MIT Technology Review)  9 Bluesky’s new AI tool is already its most blocked account—after JD Vance About 83 times as many users have blocked it as have followed it. (TechCrunch)  10 An AI agent banned from Wikipedia has lashed out in angry blogs The bot accused its human editors of “uncivil behavior.” (404 Media)   Quote of the day  “Is any of this illegal? Probably not. Is it what you’d expect from an official government app? Probably not either.”  —Security researcher Thereallo reviews the White House’s new app. One More Thing  CHANTAL JAHCHAN Inside Amsterdam’s high-stakes experiment to create fair welfare AI  When Hans de Zwart, a digital rights advocate, saw Amsterdam’s plan to have an algorithm evaluate every welfare applicant for potential fraud, he nearly fell out of his chair. He believed the system had “unfixable problems.”   Meanwhile, Paul de Koning, a consultant to the city, was excited. He saw immense potential to improve efficiencies and remove biases.  These opposing viewpoints epitomize a global debate about whether algorithms can ever make fair decisions that shape people’s lives. Read the full story.   —Eileen Guo, Gabriel Geiger, and Justin-Casimir Braun  We can still have nice things  A place for comfort, fun and distraction to brighten up your day. (Got any ideas? Drop me a line.)  + A newly authenticated Rembrandt had been hiding in plain sight for years. + This debunking of guitar legends is musical enlightenment for strummers. + Smoking into bubbles looks oddly satisfying. + The man who made the front page twice exposes the thin line between heroes and villains. 

The Download: AI health tools and the Pentagon’s Anthropic culture war 投稿を読む »

AI, Committee, ニュース, Uncategorized

Shifting to AI model customization is an architectural imperative

In the early days of large language models (LLMs), we grew accustomed to massive 10x jumps in reasoning and coding capability with every new model iteration. Today, those jumps have flattened into incremental gains. The exception is domain-specialized intelligence, where true step-function improvements are still the norm. When a model is fused with an organization’s proprietary data and internal logic, it encodes the company’s history into its future workflows. This alignment creates a compounding advantage: a competitive moat built on a model that understands the business intimately. This is more than fine-tuning; it is the institutionalization of expertise into an AI system. This is the power of customization. Intelligence tuned to context Every sector operates within its own specific lexicon. In automotive engineering, the “language” of the firm revolves around tolerance stacks, validation cycles, and revision control. In capital markets, reasoning is dictated by risk-weighted assets and liquidity buffers. In security operations, patterns are extracted from the noise of telemetry signals and identity anomalies. Custom-adapted models internalize the nuances of the field. They recognize which variables dictate a “go/no-go” decision, and they think in the language of the industry. Domain expertise in action The transition from general-purpose to tailored AI centers on one goal: encoding an organization’s unique logic directly into a model’s weights. Mistral AI partners with organizations to incorporate domain expertise into their training ecosystems. A few use cases illustrate customized implementations in practice: Software engineering and assisting at scale: A network hardware company with proprietary languages and specialized codebases found that out-of-the-box models could not grasp their internal stack. By training a custom model on their own development patterns, they achieved a step function in fluency. Integrated into Mistral’s software development scaffolding, this customized model now supports the entire lifecycle—from maintaining legacy systems to autonomous code modernization via reinforcement learning. This turns once-opaque, niche code into a space where AI reliably assists at scale. Automotive and the engineering copilot: A leading automotive company uses customization to revolutionize crash test simulations. Previously, specialists spent entire days manually comparing digital simulations with physical results to find divergences. By training a model on proprietary simulation data and internal analyses, they automated this visual inspection, flagging deformations in real time. Moving beyond detection, the model now acts as a copilot, proposing design adjustments to bring simulations closer to real-world behavior and radically accelerating the R&D loop. Public sector and sovereign AI: In Southeast Asia, a government agency is building a sovereign AI layer to move beyond Western-centric models. By commissioning a foundation model tailored to regional languages, local idioms, and cultural contexts, they created a strategic infrastructure asset. This ensures sensitive data remains under local governance while powering inclusive citizen services and regulatory assistants. Here, customization is the key to deploying AI that is both technically effective and genuinely sovereign. The blueprint for strategic customization Moving from a general-purpose AI strategy to a domain-specific advantage requires a structural rethinking of the model’s role within the enterprise. Success is defined by three shifts in organizational logic. 1. Treat AI as infrastructure, not an experiment.  Historically, enterprises have treated model customization as an ad hoc experiment—a single fine-tuning run for a niche use case or a localized pilot. While these bespoke silos often yield promising results, they are rarely built to scale. They produce brittle pipelines, improvised governance, and limited portability. When the underlying base models evolve, the adaptation work must often be discarded and rebuilt from scratch. In contrast, a durable strategy treats customization as foundational infrastructure. In this model, adaptation workflows are reproducible, version-controlled, and engineered for production. Success is measured against deterministic business outcomes. By decoupling the customization logic from the underlying model, firms ensure that their “digital nervous system” remains resilient, even as the frontier of base models shifts. 2. Retain control of your own data and models. As AI migrates from the periphery to core operations, the question of control becomes existential. Reliance on a single cloud provider or vendor for model alignment creates a dangerous asymmetry of power regarding data residency, pricing, and architectural updates. Enterprises that retain control of their training pipelines and deployment environments preserve their strategic agency. By adapting models within controlled environments, organizations can enforce their own data residency requirements and dictate their own update cycles. This approach transforms AI from a service consumed into an asset governed, reducing structural dependency and allowing for cost and energy optimizations aligned with internal priorities rather than vendor roadmaps. 3. Design for continuous adaptation. The enterprise environment is never static: regulations shift, taxonomies evolve, and market conditions fluctuate. A common failure is treating a customized model as a finished artifact. In reality, a domain-aligned model is a living asset subject to model decay if left unmanaged. Designing for continuous adaptation requires a disciplined approach to ModelOps. This includes automated drift detection, event-driven retraining, and incremental updates. By building the capacity for constant recalibration, the organization ensures that its AI does not just reflect its history, but it evolves in lockstep with its future. This is the stage where the competitive moat begins to compound: the model’s utility grows as it internalizes the organization’s ongoing response to change. Control is the new leverage We have entered an era where generic intelligence is a commodity, but contextual intelligence is a scarcity. While raw model power is now a baseline requirement, the true differentiator is alignment—AI calibrated to an organization’s unique data, mandates, and decision logic. In the next decade, the most valuable AI won’t be the one that knows everything about the world; it will be the one that knows everything about you. The firms that own the model weights of that intelligence will own the market. This content was produced by Mistral AI. It was not written by MIT Technology Review’s editorial staff.

Shifting to AI model customization is an architectural imperative 投稿を読む »

We use cookies to improve your experience and performance on our website. You can learn more at プライバシーポリシー and manage your privacy settings by clicking Settings.

Privacy Preferences

You can choose your cookie settings by turning on/off each type of cookie as you wish, except for essential cookies.

Allow All
Manage Consent Preferences
  • Always Active

Save
ja