{"id":90115,"date":"2026-05-13T16:32:27","date_gmt":"2026-05-13T16:32:27","guid":{"rendered":"https:\/\/youzum.net\/mira-muratis-thinking-machines-lab-introduces-interaction-models-a-native-multimodal-architecture-for-real-time-human-ai-collaboration\/"},"modified":"2026-05-13T16:32:27","modified_gmt":"2026-05-13T16:32:27","slug":"mira-muratis-thinking-machines-lab-introduces-interaction-models-a-native-multimodal-architecture-for-real-time-human-ai-collaboration","status":"publish","type":"post","link":"https:\/\/youzum.net\/de\/mira-muratis-thinking-machines-lab-introduces-interaction-models-a-native-multimodal-architecture-for-real-time-human-ai-collaboration\/","title":{"rendered":"Mira Murati\u2019s Thinking Machines Lab Introduces Interaction Models: A Native Multimodal Architecture for Real-Time Human-AI Collaboration"},"content":{"rendered":"

Most AI systems today work in turns. You type or speak, the model waits, processes your input, and then responds. That\u2019s the entire interaction loop. Thinking Machines Lab, an AI research lab, is arguing that this model of interaction is a fundamental bottleneck. Thinking Machines Lab team introduced a research preview of a new class of system they call interaction models<\/strong> to address it. The main idea for their research is interactivity should be native to the model itself, not bolted on as an afterthought.<\/p>\n

What\u2019s Wrong with Turn-Based AI<\/strong><\/h2>\n

If you\u2019ve built anything with a language model or voice API, you\u2019ve worked around the limitations of turn-based interaction. The model has no awareness of what\u2019s happening while you\u2019re still typing or speaking. It can\u2019t see you pause mid-sentence, notice your camera feed, or react to something visual in real time. While the model is generating, it\u2019s equally blind \u2014 perception freezes until it finishes or gets interrupted.<\/p>\n

This creates a narrow channel for human-AI collaboration that limits how much of a person\u2019s knowledge, intent, and judgment can reach the model, and how much of the model\u2019s work can be understood.<\/p>\n

To work around this, most real-time AI systems use a harness<\/strong> \u2014 a collection of separate components stitched together to simulate responsiveness. A common example is voice-activity detection (VAD)<\/strong>, which predicts when a user has finished speaking so a turn-based model knows when to start generating. This harness is made out of components that are meaningfully less intelligent than the model itself, and it precludes capabilities like proactive visual reactions, speaking while listening, or responding to cues that are never explicitly stated aloud.<\/p>\n

Thinking Machines Lab\u2019s argument is a version of the \u2018bitter lesson\u2019 in machine learning: hand-crafted systems will eventually be outpaced by scaling general capabilities. For interactivity to scale with intelligence, it must be part of the model itself. With this approach, scaling a model makes it smarter and<\/em> a better collaborator.<\/p>\n

\n
\"\"
https:\/\/thinkingmachines.ai\/blog\/interaction-models\/<\/figcaption><\/figure>\n<\/div>\n

The Architecture: Multi-Stream, Micro-Turn Design<\/strong><\/h2>\n

The system has two components working in parallel: an interaction model<\/strong> that maintains constant real-time exchange with the user, and a background model<\/strong> that handles deeper reasoning tasks asynchronously.<\/p>\n

The interaction model is always on \u2014 continuously taking in audio, video, and text and producing responses in real time. When a task requires sustained reasoning (tool use, web search, longer-horizon planning), it delegates to the background model by sending a rich context package containing the full conversation<\/strong> \u2014 not a standalone query. Results stream back as the background model produces them, and the interaction model interleaves those updates into the conversation at a moment appropriate to what the user is currently doing, rather than as an abrupt context switch. Both models share their context throughout.<\/p>\n

Think of it like one person who keeps you engaged in conversation while a colleague in the background looks something up and passes notes forward in real time.<\/p>\n

The key architectural decision enabling this is time-aligned micro-turns<\/strong>. The interaction model continuously interleaves the processing of 200ms worth of input with the generation of 200ms worth of output. Rather than consuming a complete user turn and generating a complete response, both input and output are treated as streams processed in 200ms chunks. This is what allows the model to speak while listening, react to visual cues without being prompted verbally, handle true simultaneous speech, and make tool calls and browse the web while the conversation is still in progress \u2014 weaving results back in as they arrive.<\/p>\n

Encoder-free early fusion<\/strong> is the specific design choice that makes multimodal processing work at this cadence. Rather than routing audio and video through large, separate pretrained encoders (like a Whisper-style ASR model or a standalone TTS decoder), the architecture uses minimal pre-processing. Audio signals are ingested as dMel<\/strong> and transformed via a lightweight embedding layer. Video frames are split into 40\u00d740 patches encoded by an hMLP<\/strong>. Audio output uses a flow head<\/strong> for decoding. All components are co-trained from scratch together with the transformer \u2014 there is no separately pretrained encoder or decoder at any stage.<\/p>\n

On the inference side, the 200ms chunk design creates engineering challenges. Existing LLM inference libraries aren\u2019t optimized for frequent small prefills \u2014 they carry significant per-turn overhead. Thinking Machines implemented streaming sessions<\/strong>, where the client sends each 200ms chunk as a separate request while the inference server appends chunks into a persistent sequence in GPU memory, avoiding repeated memory reallocations and metadata computations. They\u2019ve upstreamed a version of this to SGLang, the open-source inference framework. Additionally, they use a gather+gemv strategy for MoE kernels<\/strong> instead of standard grouped gemm, following prior work from PyTorch and Cursor, to optimize for the latency-sensitive shapes required by bidirectional serving.<\/p>\n

\n
\"\"
https:\/\/thinkingmachines.ai\/blog\/interaction-models\/<\/figcaption><\/figure>\n<\/div>\n

Benchmarks: Where It Stands<\/strong><\/h2>\n

The model, named TML-Interaction-Small<\/code>, is a 276B parameter Mixture-of-Experts (MoE)<\/strong> with 12B active parameters<\/strong>.<\/p>\n

The benchmark table distinguishes between Instant<\/strong> models (no extended reasoning) and Thinking<\/strong> models (with reasoning). TML-Interaction-Small<\/code> is an Instant model. Among all Instant models in the comparison, it achieves the highest score on Audio MultiChallenge APR<\/strong> at 43.4% \u2014 above GPT-realtime-2.0 (minimal) at 37.6%, GPT-realtime-1.5 at 34.7%, and Gemini-3.1-flash-live-preview (minimal) at 26.8%. The Thinking models, GPT-realtime-2.0 (xhigh) at 48.5% and Gemini-3.1-flash-live (high) at 36.1%, use extended reasoning to achieve their scores.<\/p>\n

On FD-bench v1.5<\/strong>, which measures interaction quality across user interruption, backchanneling, talking-to-others, and background speech scenarios, TML-Interaction-Small<\/code> scores 77.8 average quality \u2014 compared to 54.3 for Gemini-3.1-flash-live (minimal), 48.3 for GPT-realtime-1.5, and 47.8 for GPT-realtime-2.0 (xhigh).<\/p>\n

On FD-bench v1<\/strong> turn-taking latency, the model responds in 0.40 seconds \u2014 compared to 0.57s for Gemini, 0.59s for GPT-realtime-1.5, and 1.18s for GPT-realtime-2.0 (minimal).<\/p>\n

On FD-bench v3<\/strong>, which evaluates response quality and tool use (audio + tools combined), TML-Interaction-Small<\/code> (with background agent enabled) scores 82.8% Response Quality \/ 68.0% Pass@1 \u2014 the highest in the comparison table.<\/p>\n

\n
\"\"
https:\/\/thinkingmachines.ai\/blog\/interaction-models\/<\/figcaption><\/figure>\n<\/div>\n

Thinking Machines research team also introduced new internal benchmarks targeting capabilities that no existing model handles:<\/strong><\/p>\n