Committee

arXiv:2610.30485v1 Announce Type: cross Abstract: Personal intelligence systems are taking on increasingly complex everyday tasks, yet repeated questions, unnecessary confirmations, and unwanted outreach can turn capable assistance into interaction burden. Reducing this burden requires agents to remember how users want to be assisted. However, existing memory benchmarks focus on factual recall and personalized answers and rarely test how agents learn and act on interaction preferences. We introduce MemPABench, an interactive environment for on-policy evaluation of memory in personal agents. MemPABench poses long-horizon tasks requiring agents to decide when to clarify, confirm, or act autonomously. Four grounded simulated users hold 14 preferences across work and personal contexts. Their feedback spans 99 sessions over four months, with five event-driven changes per user testing memory updates. We evaluate preference selection, delivered behavior, and sensitivity to context and preference change. Evaluating four memory systems across four LLM backbones, we find that current memory brings limited gains in preference selection. Retrieval-based systems (i.e. Mem0, MemOS, simple RAG) do not outperform No Memory, while File-based memory improves accuracy but applies preferences beyond relevant contexts. Our analysis shows that current memory struggles to retrieve interaction preferences and adapt to preference changes. Even with the correct preference available, overriding the agent’s default interaction habits remains challenging. The environment and dataset are available at https://mempabench.com.

arXiv:2608.28886v2 Announce Type: replace Abstract: What does a generation loop gain from learning on its own verified successes? In cycles of generate, verify, select and LoRA-consolidate on online bin packing, training on value-filtered candidates shifts what the model writes on held-out variants toward value (-1.7 points of excess, p=0.008; -3.1 against a random-consolidation control, p=0.004) while the best observed candidate converges to the classic heuristic’s level and no further. A confirmation battery replicates the whole procedure three times, with fresh seeds and a never-consulted held-out set read exactly once: the mean was nearly identical in all three lineages (-2.0, -1.8, -1.9), and after aggregating within held-out variant all seven evaluable variants favored consolidation (p=0.008). The best observed candidate moved to the classic heuristic’s level, exactly (0.021028, the lowest variant’s classic level, in all three lineages and both arms); attract never went beyond its variant’s classic level, while the random control did in 7 of 24 lineage-variant cells. A matched SFT-only control shows the supervised anchor, not repulsion from bad candidates, does the concentrating (96% of candidates land exactly at the classic heuristic’s level). On far families the tails cut both ways: consolidation lowers the per-candidate rate of better-than-classic candidates (10% to 3.9%) while its larger production yields more such candidates absolutely (5 against 1, on few events). As motivation we report the inference-time ledger that led here: a model-written schematic recap buys judged document integration and nothing buys development; a verifier written into the stream is imitated, 16.4 fabricated verdict lines per notebook. In this arrangement (an 8-bit model in MLX, five cycles), mean quality among valid candidates can be bought and replicated; under attraction the observed best goes to the classic and, so far, never beyond it.

Earlier this week, I attended a screening of You Can See Everything, the buzzy new documentary about the disgraced former biotech CEO Elizabeth Holmes.  I went in not really knowing what to expect. I’d avoided spoilers (and won’t be giving anything away here, either!). I was wondering what new information this film could reveal, given how familiar many of us are with the Theranos story thanks to years of media coverage, the best-selling book Bad Blood, and the Hulu series The Dropout, starring Amanda Seyfried. Like many others who have seen the film, I came away feeling unsettled. But not just about Holmes’s fraud and the somewhat disturbing interactions between the subjects. The film reignited a concern I’ve had about covering science for most of my career: Despite our best efforts, we never can actually see everything. In case you’re rusty on the details, Elizabeth Holmes dropped out of Stanford University in 2003 to run Theranos, a diagnostics company she founded, when she was just 19. She went on to claim that her company had invented a way to run over 100 diagnostic tests on just a few drops of blood. Holmes raised hundreds of millions of dollars for Theranos, which was valued at $9 billion in 2014. Forbes named her the youngest self-made female billionaire—she was worth $4.5 billion when she was just 30 years old. Things started to unravel in 2015, when an investigation by the Wall Street Journal raised concerns by former Theranos employees that the tests didn’t work. Some also said that the company’s device had not been tested properly, and that blood samples had been analyzed on machines from other companies instead. MIT Technology Review named it one of the biggest technology failures of 2015. Forbes had revised Holmes’s worth to $0 by 2016. That year, a damning government report said that the company’s lab was struggling with basic procedures and gave patients results even when they knew they could be wrong. In 2018, Theranos, Holmes, and former company president Ramesh Balwani were charged with “massive fraud.” In 2022, she was sentenced to a little over 11 years in prison for defrauding investors. She began serving her sentence in May 2023. It was around a month before this that filmmakers Nathan Fielder, Lance Oppenheim, and their colleagues began documenting Holmes and her partner, Billy Evans, at home with their two young children. I’m not giving anything away when I say it’s an extremely intimate portrayal—at least of how the two adults interact with Fielder, who is also a central character in the film. As you might know if you’ve seen the trailer, there are plenty of super close-ups. I felt as if I spent most of the film staring straight into the faces of those people. And almost the whole way through, they all seemed to be trying to figure each other out. Holmes is known for her dishonesty, but she invited cameras into her home in an apparent bid to reframe the narrative and position herself as an honest person with an unusual mind. Will anyone believe her? I couldn’t help feeling some empathy for Holmes, a mother we see preparing to leave her two kids for a lengthy prison sentence. And even though I have long known the facts, Holmes is clearly an effective manipulator. You can see her plant the seeds of doubt in Fielder’s mind. At times, I felt she was convincing me, sitting in my cinema seat. Maybe she really didn’t mean to lie? But then I remembered how other journalists felt after being taken in by her claims. “I blew it,” Fortune’s Roger Parloff reflected in 2015, a year after profiling Holmes and her company. “I’m pretty sure she lied to me,” STAT’s Matthew Herper wrote recently, having interviewed Holmes in 2015. Even the former employee turned whistleblower Tyler Shultz, who knew the product didn’t work, said that he could “have a five-minute conversation with her and walk away feeling like my work was changing the world,” he said in 2022. I didn’t cover Theranos in the 2010s. If I’d interviewed Holmes, would I have fallen for it all? One of the early red flags in the Theranos case was the lack of published, peer-reviewed data. Much of the company’s work was classified as a “trade secret.” Scientists tend to publish their work in journals—a slow process that involves having the work evaluated by often anonymous experts in the field, who provide feedback, highlight questionable conclusions, and often ask for clarification. Peer review is meant to add rigor to the publication process. You should be able to trust published scientific papers. Except that sometimes you can’t. Scientists aren’t perfect. Some will exaggerate their results or inflate their importance. Some manipulate data to serve a hypothesis. Sometimes these infractions are caught during peer review, but not always. There are plenty of examples of scientific findings that could never be replicated by other labs and ended up collapsing under the weight of real evidence. Many papers ultimately end up with corrections. Some are withdrawn—10,000 research papers were retracted in 2023. At any rate, there’s been a shift to sharing research in other ways. Today, many scientists opt to publish their work at preprint servers like arXiv, medRxiv, and bioRxiv before they submit them to established journals. Those outlets bypass peer review. So do company press releases, social media posts, and many conference presentations. And I get a lot of my stories from conversations and interviews. I’ll admit I’ve looked at a scientific finding and thought: A scientist has just said they’ve done that. How can I be sure they really have? I like to think that I hold myself to high journalistic standards here, as does everyone at MIT Technology Review. I would never publish a story about some new scientific development without doing my own form of peer review, and asking scientists—ideally people I know and trust—what they think of a new claim.  But the film is a stark

How much do you know about permafrost? Maybe, like me, you’ve been aware that this layer of frozen soil is thawing, that it’s important in the global puzzle of climate change … and that’s about the extent of it.   The great news for both of us is that my colleague James Temple just published two in-depth stories about permafrost. His reporting took him to the boreal forests of Alaska, where he helicoptered into the wilderness to learn more about how this material could affect our climate both now and in the future. I’d recommend you give both of James’s stories a read. There’s this feature about efforts to tackle the intensifying fires that are threatening permafrost, as well as this look at how researchers are trying to understand the complicated tangle of potential climate effects. Here are three things I took away. Permafrost is a massive blind spot for climate change Permafrost is the frozen layer of soil that’s under about 15% of land around the Northern Hemisphere. While technically material needs to be frozen for only two consecutive years to count as permafrost, most of our current permafrost formed during the last ice age. That means there’s organic material that’s been frozen, and essentially locked away, for thousands of years. But as global temperatures tick up and this layer thaws, microbes are able to break down that organic matter, releasing carbon dioxide and methane—powerful greenhouse gases that drive climate change. Those emissions form a feedback loop, accelerating warming further. The steady rise in global temperatures has already shrunk the footprint of the Arctic permafrost by about 12% since 1850, and the pace has been picking up in the last 50 years. The wild thing is, only a minority of the models that have been used in UN climate reports actually take into account the gradual loss of permafrost. And none of them include the abrupt thaw and collapse that can happen after events like heat waves and fires (more on those fires in a moment). There are efforts to make sure future UN reports reflect the growing science, but for now, all this means that we’re probably underestimating how much and how fast the planet is going to warm. Wildfires can speed up permafrost loss Fire is a natural part of boreal forests’ life cycle, but they’re getting more intense because of climate change. And crucially, fires can accelerate the thawing of permafrost. By the end of the century, this effect could release the equivalent of 80 billion metric tons of carbon dioxide. Today, the policy across remote parts of Canada and Alaska is generally to let wildfires burn as long as they’re not threatening human habitats. But some researchers say we should explore fighting these fires to save the carbon trapped in the forests and the permafrost beneath them.   Researchers at the Woodwell Climate Research Center are developing a map of burn risk and carbon vulnerability across Alaska, essentially highlighting areas where fires are most likely to spread and which areas are home to certain types of permafrost that would release the most emissions. (Yes, apparently there are different types of permafrost. Yedoma is a particularly deep, carbon-dense form from the late Pleistocene era.) That’s all just the beginning—once we figure out potential targets, the question is what to do about it. Some suggest setting more prescribed fires, something Indigenous groups have done for thousands of years, partly to reduce the danger of massive blazes. Or resources could go to early detection and efforts to put out fires before they grow. The carbon cycle of permafrost is incredibly complicated There’s about 1.5 trillion tons of carbon locked away in permafrost, about twice the amount in the atmosphere. We know that permafrost thawing can release greenhouse-gas emissions and drive climate change. But there are complications.   One particularly big wrinkle: Other effects from permafrost thawing and fire recovery might counteract the carbon lost from those events. For example, as temperatures warm, plants can gradually move north and take hold where they couldn’t have before. Those plants soak up carbon from the atmosphere as they grow, at least partly locking away what was emitted. Researchers are working to better understand this ecosystem. And the more we learn, the more complicated the picture looks. How much carbon is lost from permafrost, and how much a given area will recover its carbon, depend on a huge array of factors. How severe are the fires that have accelerated local thawing? What’s the local climate and environment? And what kind of permafrost is at stake? The higher we push the planet’s temperature, the more we mess with complicated natural phenomena that we’re only beginning to understand.  This article is from The Spark, MIT Technology Review’s weekly climate newsletter. To receive it in your inbox every Wednesday, sign up here. 

Enterprise AI can generate recommendations, automate tasks, and help organizations make decisions faster. But without an understanding of how a business actually operates, even powerful models can miss the constraints and relationships that determine whether a decision is the right one. As companies look to move AI from experimentation into everyday operations, the challenge ahead is giving technology the context to act usefully within complex business processes. For Devavrat Shah, chief scientist of enterprise AI at Celonis, that context is central to what he calls “decision intelligence.” AI agents can support individual tasks within larger decision-making processes, he argues, but they need to understand the role each task plays and the business conditions surrounding it. “You can’t just do that in a vacuum,” he says. Without the right operational context, agents risk overlooking important constraints, making costly mistakes, or consuming excessive computing resources as they try to work out what they need to know. The solution begins with connecting AI to the processes and data enterprises already use. A contextual layer can help agents understand workflows, operate within defined guardrails, and produce recommendations that people can review and approve. Shah advises organizations to start with a specific business decision, identify its most difficult steps, and test whether AI can improve them. Measuring gains in speed and value can then help determine whether an application is worth scaling. This approach could also change how employees spend their time. Automating repetitive, manual work and coordinating tasks across teams could free people to focus on more complex strategic decisions. Realizing that benefit, however, will require investment in workforce education, so employees can understand AI outputs and their implications instead of just handing decisions over to agents. Looking to the future, Shah sees opportunities extending from routine operational tasks to enterprise-wide planning in areas such as finance, workforce management, marketing, and sales. With the right context, AI could help organizations evaluate options and make consequential decisions more quickly. As Shah puts it, “Now, maybe [resource planning] should not take that much time, and people are actually making the right choices quickly.” This episode of Business Lab is produced in partnership with Celonis. Full Transcript Megan Tatum: From MIT Technology Review, I’m Megan Tatum, and this is Business Lab, the show that helps business leaders make sense of new technologies coming out of the lab and into the marketplace. This podcast is produced in partnership with Celonis. Now, enterprise AI is at something of a crossroads. After years of experimentation, demos, and pilots, leadership teams want clear proof that scaling up will positively impact their bottom line, but many are missing an integral part of the puzzle. By acting without operational context, enterprise AI and AI agents are performing at a significant deficit. They’re applying generalized principles to specialized scenarios and coming up short, leaving business leaders understandably unconvinced and AI’s potential unrealized. Two words for you: context matters. My guest today is Devavrat Shah, chief scientist of enterprise AI at Celonis. Welcome, Devavrat. Devavrat Shah: Hello, Megan. Thanks for having me here. Megan: Thank you so much for being here with us today. Devavrat, just to start, why is it so crucial that organizations are thinking about operational context as it relates to enterprise AI right now? What makes it such a critical part of that transition I was talking about from experimentation to scale? Devavrat: It’s a fantastic question. And as you correctly noted, we’re at the cusp of becoming experimental to mainstream as far as AI is concerned. As you know, I’ve been teaching AI at MIT as a professor for more than the last two decades, so it’s no surprise that I’m very bullish about AI’s future in enterprises. The way I think about where AI can make the biggest difference in enterprises is in decision making, helping enterprise make decisions, or putting it other way, how do we bring decision intelligence to enterprises? As we think about decisions, how do we make decisions today? Take MIT Tech Review, take Celonis, take MIT, take any large organization. We have different types of decisions we want to make in very broad strokes. There are decisions which are strategic decisions like episodic, other decisions which are operational that need to be done with a high frequency. In either of them, we have some processes in place. You do this first, you do that first, you do that first, you do that first. Many of us help coordinate those things. Many of us play roles within it, what Jensen Huang calls task. And once you get those things done step by step, then you reach decision. Now, it’s those tasks, or what I would call roles, that’s where AI can bring in an atomic manner agentic power so that we can at least parts of decisions, if not all of the decisions, we can augment support and maybe sometimes when it’s a painful automate. And to get to that, to know which role to do, which task to do, we really need to know the context. You can’t just do that in a vacuum. And that’s why context with AI put together in the right way would bring what I would call decision intelligence to enterprise. And that’s where I think there’ll be the biggest transformation that would come. Megan: Right. To drill down into that into a bit more detail, we know many organizations are, as you say, embracing identic AI right now with this potential for increasingly autonomous decision making. If context isn’t introduced in parallel, what are the commercial risks at play? Devavrat: Fantastic. Let’s just take the setting I was describing. We are trying to automate a role or support a role or augment a role depending on the type of decision, depending upon type of task. And let’s for a second imagine that all these, as they’re called, frontier labs have built these amazing models and amazing AI infrastructure so that everything that’s happened in the universe and