Committee

The mouse pointer has sat at the center of personal computing for more than half a century. It tracks cursor position. It registers clicks. Beyond that, it does almost nothing. Google DeepMind researchers outlined a set of experimental principles and demos for an AI-enabled pointer that goes considerably further: one that understands not just where you are pointing, but what you are pointing at and why it matters. The system is powered by Gemini and is currently in the experimental stage. Two demos are live in Google AI Studio today: one for editing an image and one for finding places on a map, both operable by pointing and speaking. A deeper integration called Magic Pointer is also rolling out inside Chrome, and a further integration is planned for Googlebook, Google’s new line of Gemini-powered laptops announced this week. https://deepmind.google/blog/ai-pointer/ What DeepMind is Targeting The frustration DeepMind researchers are addressing is a familiar one for anyone who has tried to use an AI assistant while already in the middle of work. Because a typical AI tool lives in its own window, users need to drag their world into it. The research team wants the opposite — intuitive AI that meets users across all the tools they use, without interrupting their flow. In practice, today’s AI workflow often looks like this: you are working inside a document or a browser tab, you spot something you want to ask about, you switch to a chat interface, you re-describe what you were looking at, you run the query, and you paste the result back. This maps to a concrete technical gap: current LLM interfaces are largely text-in, text-out. They have no awareness of the screen state around them. The AI-enabled pointer is an attempt to close that gap by giving the model real-time visual and semantic context derived from cursor position and hover state — without requiring users to manually serialize that context into a written prompt. Four interaction principles DeepMind researchers have developed four principles that together shift the hard work of conveying context and intent from the user to the computer, replacing text-heavy prompts with simpler, more intuitive interactions. The first is Maintain the flow. AI capabilities should work across all apps, not force users into ‘AI detours’ between them. The prototype AI-enabled pointer is available wherever the user is working. For example, they could point at a PDF and request a bullet-point summary to paste directly into an email, hover over a table of statistics and request a pie chart version, or highlight a recipe and ask for all the ingredients doubled. This is a direct architectural stance: instead of building AI assistance as a sidecar application, the capability lives at the pointer level and is present in whichever tool the user is already working in. The second is Show and tell. Current AI models demand precise instructions. To get a good response, a user has to write a detailed prompt. An AI-enabled pointer would streamline this process by smoothly capturing the visual and semantic context around the pointer, letting the computer ‘see’ and understand what’s important to the user. In the experimental system, just point, and the AI knows exactly which word, paragraph, part of an image, or code block the user needs help with. From a technical standpoint, this means the system treats cursor hover state and the surrounding UI content as structured model inputs — comparable to how multimodal models process image and text together, except here the visual region is dynamically cropped and contextualized in real time around a moving cursor. The third is Embrace the power of ‘This’ and ‘That‘. In everyday interactions with each other, humans rarely speak in long, detailed paragraphs. We might say, ‘Fix this’, ‘Move that here’, or ‘What does this mean?’ — while relying on physical gestures and our shared context to fill in any gaps in understanding. An AI system that understands this combination of context, pointing and speech would allow users to make complex requests in natural shorthand, no fiddly prompting required. The name of the principle is deliberate: deictic language (words like ‘this’ and ‘that’ that depend on physical reference to carry meaning) is how humans naturally communicate when they can point at something. The AI-enabled pointer is designed to handle exactly that class of instruction without needing the user to spell out what “this” refers to. The fourth is Turn pixels into actionable entities. For decades, computers have only tracked where we are pointing. AI can now also understand what the user is pointing at. This transforms pixels into structured entities, such as places, dates, and objects, that users can interact with instantly. A photo of a scribbled note becomes an interactive to-do list; a paused frame in a travel video becomes a booking link for that cool-looking restaurant. For ML engineers, this is the most technically substantive of the four principles. It describes an entity extraction step that happens at inference time on whatever visual content is under the cursor — converting raw pixel regions into typed, actionable objects rather than leaving them as unstructured screen content. Where it is going Google DeepMind is now integrating these principles to reimagine pointing in Chrome and the new Googlebook laptop experience. Starting now, instead of writing a complex prompt, users can use their pointer to ask Gemini in Chrome about the part of the webpage they care about. For example, selecting a few products on a page and asking to compare them, or pointing to where they want to visualize a new couch in their living room. Key Takeaways Google DeepMind introduces experimental demos of an AI-enabled mouse pointer powered by Gemini that captures visual and semantic context around the cursor — no manual prompting required. The system is built on four principles: Maintain the flow, Show and tell, Embrace the power of “This” and “That”, and Turn pixels into actionable entities. “Turn pixels into actionable entities” is the key technical idea — the pointer converts on-screen

Most AI systems today work in turns. You type or speak, the model waits, processes your input, and then responds. That’s the entire interaction loop. Thinking Machines Lab, an AI research lab, is arguing that this model of interaction is a fundamental bottleneck. Thinking Machines Lab team introduced a research preview of a new class of system they call interaction models to address it. The main idea for their research is interactivity should be native to the model itself, not bolted on as an afterthought. What’s Wrong with Turn-Based AI If you’ve built anything with a language model or voice API, you’ve worked around the limitations of turn-based interaction. The model has no awareness of what’s happening while you’re still typing or speaking. It can’t see you pause mid-sentence, notice your camera feed, or react to something visual in real time. While the model is generating, it’s equally blind — perception freezes until it finishes or gets interrupted. This creates a narrow channel for human-AI collaboration that limits how much of a person’s knowledge, intent, and judgment can reach the model, and how much of the model’s work can be understood. To work around this, most real-time AI systems use a harness — a collection of separate components stitched together to simulate responsiveness. A common example is voice-activity detection (VAD), which predicts when a user has finished speaking so a turn-based model knows when to start generating. This harness is made out of components that are meaningfully less intelligent than the model itself, and it precludes capabilities like proactive visual reactions, speaking while listening, or responding to cues that are never explicitly stated aloud. Thinking Machines Lab’s argument is a version of the ‘bitter lesson’ in machine learning: hand-crafted systems will eventually be outpaced by scaling general capabilities. For interactivity to scale with intelligence, it must be part of the model itself. With this approach, scaling a model makes it smarter and a better collaborator. https://thinkingmachines.ai/blog/interaction-models/ The Architecture: Multi-Stream, Micro-Turn Design The system has two components working in parallel: an interaction model that maintains constant real-time exchange with the user, and a background model that handles deeper reasoning tasks asynchronously. The interaction model is always on — continuously taking in audio, video, and text and producing responses in real time. When a task requires sustained reasoning (tool use, web search, longer-horizon planning), it delegates to the background model by sending a rich context package containing the full conversation — not a standalone query. Results stream back as the background model produces them, and the interaction model interleaves those updates into the conversation at a moment appropriate to what the user is currently doing, rather than as an abrupt context switch. Both models share their context throughout. Think of it like one person who keeps you engaged in conversation while a colleague in the background looks something up and passes notes forward in real time. The key architectural decision enabling this is time-aligned micro-turns. The interaction model continuously interleaves the processing of 200ms worth of input with the generation of 200ms worth of output. Rather than consuming a complete user turn and generating a complete response, both input and output are treated as streams processed in 200ms chunks. This is what allows the model to speak while listening, react to visual cues without being prompted verbally, handle true simultaneous speech, and make tool calls and browse the web while the conversation is still in progress — weaving results back in as they arrive. Encoder-free early fusion is the specific design choice that makes multimodal processing work at this cadence. Rather than routing audio and video through large, separate pretrained encoders (like a Whisper-style ASR model or a standalone TTS decoder), the architecture uses minimal pre-processing. Audio signals are ingested as dMel and transformed via a lightweight embedding layer. Video frames are split into 40×40 patches encoded by an hMLP. Audio output uses a flow head for decoding. All components are co-trained from scratch together with the transformer — there is no separately pretrained encoder or decoder at any stage. On the inference side, the 200ms chunk design creates engineering challenges. Existing LLM inference libraries aren’t optimized for frequent small prefills — they carry significant per-turn overhead. Thinking Machines implemented streaming sessions, where the client sends each 200ms chunk as a separate request while the inference server appends chunks into a persistent sequence in GPU memory, avoiding repeated memory reallocations and metadata computations. They’ve upstreamed a version of this to SGLang, the open-source inference framework. Additionally, they use a gather+gemv strategy for MoE kernels instead of standard grouped gemm, following prior work from PyTorch and Cursor, to optimize for the latency-sensitive shapes required by bidirectional serving. https://thinkingmachines.ai/blog/interaction-models/ Benchmarks: Where It Stands The model, named TML-Interaction-Small, is a 276B parameter Mixture-of-Experts (MoE) with 12B active parameters. The benchmark table distinguishes between Instant models (no extended reasoning) and Thinking models (with reasoning). TML-Interaction-Small is an Instant model. Among all Instant models in the comparison, it achieves the highest score on Audio MultiChallenge APR at 43.4% — above GPT-realtime-2.0 (minimal) at 37.6%, GPT-realtime-1.5 at 34.7%, and Gemini-3.1-flash-live-preview (minimal) at 26.8%. The Thinking models, GPT-realtime-2.0 (xhigh) at 48.5% and Gemini-3.1-flash-live (high) at 36.1%, use extended reasoning to achieve their scores. On FD-bench v1.5, which measures interaction quality across user interruption, backchanneling, talking-to-others, and background speech scenarios, TML-Interaction-Small scores 77.8 average quality — compared to 54.3 for Gemini-3.1-flash-live (minimal), 48.3 for GPT-realtime-1.5, and 47.8 for GPT-realtime-2.0 (xhigh). On FD-bench v1 turn-taking latency, the model responds in 0.40 seconds — compared to 0.57s for Gemini, 0.59s for GPT-realtime-1.5, and 1.18s for GPT-realtime-2.0 (minimal). On FD-bench v3, which evaluates response quality and tool use (audio + tools combined), TML-Interaction-Small (with background agent enabled) scores 82.8% Response Quality / 68.0% Pass@1 — the highest in the comparison table. https://thinkingmachines.ai/blog/interaction-models/ Thinking Machines research team also introduced new internal benchmarks targeting capabilities that no existing model handles: TimeSpeak — Tests whether the model initiates speech at user-specified times with correct content.

Most

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. A plan to make drugs in orbit is going commercial A startup called Varda Space Industries is betting that the future of pharmaceuticals lies in orbit. The company has signed a deal with United Therapeutics to test whether drugs crystallize differently in microgravity, potentially creating improved versions with new properties. The idea sounds futuristic, but falling launch costs and reusable rockets are making space-based manufacturing seem increasingly plausible. Varda says the partnership could mark an important step toward building products in orbit for use back on Earth. Discover how space could become the next frontier for drug development. —Antonio Regalado MIT Technology Review Narrated: NASA is building the first nuclear reactor-powered interplanetary spacecraft. How will it work? Just before Artemis II began its historic slingshot around the moon, NASA revealed an even grander space travel plan. By the end of 2028, the agency aims to fly a nuclear reactor-powered interplanetary spacecraft to Mars. A successful mission would herald a new era in spaceflight—and might just give the US the edge in the race against China. But the project remains shrouded in mystery. MIT Technology Review picked the brains of nuclear power and propulsion experts to find out how the nuclear-powered spacecraft might work. —Robin George Andrews This is our latest story to be turned into an MIT Technology Review Narrated podcast, which we publish each week on Spotify and Apple Podcasts. Just navigate to MIT Technology Review Narrated on either platform, and follow us to get all our new content as it’s released. The must-reads I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology. 1 Sam Altman claims Elon Musk tried to seize control of OpenAIAltman said Musk initially wanted 90% of the equity. (AFP)+ And that control should go to his children when he dies. (BBC)+ Altman also accused Musk of twice trying to end its non-profit status. (NPR)+ Musk’s motivations for the suit are under scrutiny. (MIT Technology Review) 2 Google and SpaceX are in talks to launch data centers into orbitSpaceX could join Suncatcher, Google’s orbital data center project. (WSJ $)+ The project’s first launch is slated for early 2027. (Guardian)+ Anthropic and SpaceX have also discussed orbital data centers. (Wired $)+ But there are a few hurdles to overcome. (MIT Technology Review)  3 Jensen Huang has joined Donald Trump’s high-stakes mission to ChinaNvidia is lobbying to sell its AI chips in the country. (Bloomberg $)+ Elon Musk and Tim Cook are also on the trip. (CNBC)+ But a tech rivalry and distrust have sapped hopes for big deals. (Reuters $) 4 ICE agents have a list of 20 million people on their iPhones, thanks to PalantirAn ICE official said Palantir is speeding up raids and arrests. (404 Media)+ ICE has also used facial recognition and Paragon spyware. (TechCrunch) 5 Defense tech firm Anduril just doubled its valuation to over $60 billionIn a $5 billion funding round led by Thrive Capital and a16z. (FT $)Anduril, which makes AI-backed weapons, may go public next year. (NYT $) 6 Meta employees are protesting computer-tracking at workFlyers posted at offices are urging staff to oppose the program. (Reuters $)+ Meta plans to track workers’ clicks and keystrokes to train AI. (CNBC) 7 OpenAI is facing another wrongful death lawsuit over ChatGPT medical adviceThe chatbot’s tips allegedly led to a teenager’s overdose. (Ars Technica) 8 The Canvas learning platform has paid hackers to delete stolen student dataIt caved to ransomware demands after the biggest-ever edtech breach. (BBC) 9 Scientific researchers are thinking twice about using AIDue to price hikes, usage limitations, and unreliable outputs. (Nature) 10 The latest AI compute solution? Putting data centers in your homeHardware hosts get subsidized electricity and internet. (Ars Technica) Quote of the day “Mr Musk did try to kill it.” —Sam Altman claims that Elon Musk tried to destroy rather than protect OpenAI’s non-profit operations, the Guardian reports. One More Thing YOSHI SODEOKA Why does AI hallucinate? Chatbot fails are now a familiar meme. Meta’s short-lived scientific chatbot generated wiki articles about the history of bears in space. Lawyers have submitted court documents filled with legal citations fabricated by ChatGPT. Air Canada was ordered to honor a refund policy invented by its customer service chatbot. This tendency to make things up—known as hallucination—is one of the biggest obstacles holding chatbots back from more widespread adoption. Here’s why they do it—and why we still can’t fix it. —Will Douglas Heaven This story is part of MIT Technology Review Explains, our series untangling the complex, messy world of technology to help you understand what’s coming next. You can read more from the series here.  We can still have nice things A place for comfort, fun, and distraction to brighten up your day. (Got any ideas? Drop me a line.) + A historian has unearthed the etymology of every single dinosaur name.+ Humus on the moon is getting closer to reality after scientists grew chickpeas in lunar soil.+ Witness the patience of a master paper artist in this gallery of intricate, handmade sculptures.+ Want to tell the time alphabetically? Me neither, but this cursed clock is an intriguing reason to try.

Large language models (LLMs) now power everything from customer service bots to autonomous coding agents.

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. Three things in AI to watch, according to a Nobel-winning economist A few months before he won the Nobel Prize in economics in 2024, Daron Acemoglu published a paper that earned him few fans in Silicon Valley. He argued that AI would give only a small boost to US productivity and would not eliminate the need for human work. Two years later, Acemoglu’s measured take has not caught on. The technology has advanced quite a bit since his cautious predictions, but the data is still largely on his side.  MIT Technology Review spoke with him to understand if any of the latest developments have changed his thesis. Here are the three things Acemoglu is paying closest attention to in AI right now. —James O’Donnell This story is from The Algorithm, our weekly newsletter giving you the inside track on all things AI. Sign up to receive it in your inbox every Monday.  The case for fixing everything Stewart Brand, the counterculture icon and tech industry legend, considers maintenance a “civilizational” act. His new book argues that taking responsibility for maintaining something, whether a motorcycle, a monument, or the planet, can be radical. Brand argues that maintainers haven’t gotten the laurels they deserve—and he’s right. Yet his vision of maintenance often feels solitary: profound, but more about personal fulfillment than tending to a shared world or making it better. Read the full review of his handsome new book, Maintenance: Of Everything, Part One. —Lee Vinsel Lee Vinsel is an associate professor of science, technology, and society at Virginia Tech, a cofounder of The Maintainers, and the host of Peoples & Things, a podcast about human life with technology. This story is from the latest edition of our print magazine, which is all about nature. Subscribe now to read the full issue and receive future print copies once they land. The must-reads I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology. 1 The first zero-day exploit built by AI has been discoveredGoogle spotted and stopped the attempted “mass exploitation event.” (CNBC)+ The hackers used AI to discover an unknown bug. (NYT $)+ AI-powered hacking has exploded into an industrial-scale threat. (Guardian)+ New tools are simplifying online crime. (MIT Technology Review) 2 OpenAI just launched its answer to Claude MythosDaybreak patches vulnerabilities before attackers find them. (The Verge)+ Sam Altman said it will “continuously secure software.” (Gizmodo)+ It will rival Anthropic’s Claude Mythos, which arrived a month ago. (BBC)+ OpenAI is allowing wider access to its cyber models than Anthropic. (CNBC) 3 Trump is heading to China to spread the gospel of American techWhile taking cues from Beijing’s more stringent approach. (Guardian)+ But investors want Trump and Xi to stay out of AI’s way. (Reuters $)+ Elon Musk and Tim Cook are joining him on the trip this week. (BBC) 4 Ilya Sutskever has testified on Sam Altman’s “pattern of lying”OpenAI co-founder Sutskever took the stand in the Altman v. Musk trial. (BI)+ He said he spent a year gathering proof of Altman’s dishonesty. (Reuters $)+ But he also added to OpenAI’s defense. (Wired $)+ While Satya Nadella called attempts to remove Altman “amateur city.” (FT $)+ Here’s what happened last week in the trial. (MIT Technology Review) 5 A new hantavirus vaccine is in the worksModerna and Korea University are developing an mRNA vaccine. (Wired $)+ Here’s what you need to know about the cruise ship outbreak. (MIT Technology Review) 6 Texas has sued Netflix over alleged data harvesting and “addictive” designAG Ken Paxton accuses Netflix of secretly collecting and selling user data. (Quartz)+ And spying on children while deliberately fostering addiction. (Guardian) 7 A data center guzzled 30 million gallons of water—and no one noticedThe curious case serves as a warning for other data center projects. (Ars Technica) 8 Europe is reportedly selling spyware to human rights abusersEU states allegedly sold the tech to countries violating rights. (Bloomberg $) 9 The US government’s AI vetting announcement has mysteriously vanishedIt had detailed a security test agreement with Google, xAI, and Microsoft. (Gizmodo) 10 Amazon staff are using AI for pointless tasks just to inflate usage scoresIn a bid to impress managers. (FT $) + An AI expert says we should stop using AI so much. (MIT Technology Review) Quote of the day “This is like the cheating husband complaining about the cheating wife.”  —Anupam Chander, a professor of law and technology at Georgetown Law School, tells the New York Times that Elon Musk’s hypocrisy over OpenAI becoming a for-profit company will undermine his courtroom battle with Sam Altman. One More Thing STUART BRADFORD How sounds can turn us on to the wonders of the universe For decades, astronomy has relied on visual information to make sense of the cosmos: images, charts, and graphs. Now, some researchers are trying something different: listening to the universe. Using sonification, the process of turning information into sound, they’re helping blind and visually impaired researchers explore the cosmos—and even uncover patterns that might otherwise go unnoticed. The approach is spreading beyond astronomy into fields like climate science, navigation, and education. Discover how sound could make science more accessible—and even more revealing. —Corey S. Powell We can still have nice things A place for comfort, fun, and distraction to brighten up your day. (Got any ideas? Drop me a line.) + This musical mashup beautifully blends LCD Soundsystem with Twin Peaks.+ Match your speculative ideas to sci-fi stories with the Extrapolated Futures Archive.+ A live-action animation Coyote vs. ACME is coming soon—and the first trailer just dropped.+ Want to surf elsewhere in the galaxy? Here’s what it would be like to catch waves on distant planets.