Uncategorized

arXiv:2506.09886v2 Announce Type: replace Abstract: Detecting hallucinations in large language models (LLMs) is critical for their safety in many applications. Without proper detection, these systems often provide harmful, unreliable answers. In recent years, LLMs have been actively used in retrieval-augmented generation (RAG) settings. However, hallucinations remain even in this setting, and while numerous hallucination detection methods have been proposed, most approaches are not specifically designed for RAG systems. To overcome this limitation, we introduce a hallucination detection method based on estimating the distances between the distributions of prompt token embeddings and language model response token embeddings. The method examines the geometric structure of token hidden states to reliably extract a signal of factuality in text, while remaining friendly to long sequences. Extensive experiments demonstrate that our method achieves state-of-the-art or competitive performance. It also has transferability from solving the NLI task to the hallucination detection task, making it a fully unsupervised and efficient method with a competitive performance on the final task.

arXiv:2602.21368v1 Announce Type: cross Abstract: Given a black-box AI system and a task, at what confidence level can a practitioner trust the system’s output? We answer with a reliability level — a single number per system-task pair, derived from self-consistency sampling and conformal calibration, that serves as a black-box deployment gate with exact, finite-sample, distribution-free guarantees. Self-consistency sampling reduces uncertainty exponentially; conformal calibration guarantees correctness within 1/(n+1) of the target level, regardless of the system’s errors — made transparently visible through larger answer sets for harder questions. Weaker models earn lower reliability levels (not accuracy — see Definition 2.4): GPT-4.1 earns 94.6% on GSM8K and 96.8% on TruthfulQA, while GPT-4.1-nano earns 89.8% on GSM8K and 66.5% on MMLU. We validate across five benchmarks, five models from three families, and both synthetic and real data. Conditional coverage on solvable items exceeds 0.93 across all configurations; sequential stopping reduces API costs by around 50%.

When a company claims to have created what’s essentially the holy grail of batteries, there are bound to be some questions. Interest has been swirling since Donut Lab, a Finnish company, announced last month that it had a new solid-state battery technology, one that was ready for large-scale production. The company said its batteries can charge super-fast and have a high energy density that would translate to ultra-long-range EVs. What’s more, it claimed the cells can operate safely in the extreme heat and cold, contain “green and abundant materials,” and would cost less than lithium-ion batteries do today. It sounded amazing—this sort of technology could transform the EV industry. But many quickly wondered if it was all too good to be true. Now, Donut Lab is releasing a series of videos that it says will prove its technology has the secret sauce. Let’s dig into why this company is making news, why many experts are skeptical, and what it all means for the battery industry right now. Solid-state batteries could deliver the next generation of EVs. In place of a liquid electrolyte (the material that ions move through inside a battery), the cells use a solid material, so they can be more compact. That means a significantly longer range, which could get more people excited to drive EVs. The problem is, getting these batteries to work and making them at the large scale required for the EV industry hasn’t been a simple task. Some of the world’s most powerful automakers and battery companies have been trying for years to get the technology off the ground. (Toyota at one point said it would have solid-state batteries in cars by 2020. Now it’s shooting for 2027 or 2028.) While it’s been a long time coming, it does feel as if solid-state batteries are closer than ever. Much of the progress so far has been on semi-solid-state batteries, which use materials like gels for electrolytes. But some companies, including several in China, are getting closer to true solid state. The world’s largest battery company, CATL, plans to manufacture small quantities in 2027. Another major Chinese automaker, Changan, plans to start testing installation of all-solid-state batteries in vehicles this year, with mass production expected to begin next year. Still, Donut Lab surprised the battery industry when, in a video released in early January ahead of the Consumer Electronics Show in Las Vegas, the company claimed it would put the world’s first all-solid-state battery into production vehicles. One of the splashiest claims in the announcement was that cells would have an energy density of 400 watt-hours per kilogram (the top commercial lithium-ion batteries today sit at about 250 to 300 Wh/kg). It was also claimed that the cells could charge in as little as five minutes, last 100,000 cycles, and retain 99% of capacity at high and low temperatures—while costing less than lithium-ion cells and being made from “100% green and abundant materials with global availability.” Many experts were immediately skeptical. “In the solid-state field, the technical barriers are very high,” said Shirley Meng, a professor of molecular engineering at the University of Chicago, when I spoke with her last month. She’d recently attended CES and visited Donut Lab’s booth. “They had zero demo, so I don’t believe it,” she says. “Call me conservative, but I would rather be careful than be sorry later.” “It’s one of those things where nobody knows—they’ve never heard of it,” said Eric Wachsman, a professor at the University of Maryland and cofounder of the solid-state battery company Ion Storage Systems, in a January interview. “They came out of nowhere.” Donut Lab has shared very little about what, exactly, this technology might be. It’s not uncommon for battery companies (or any startup, for that matter) to be quiet about technical details before they can get patents filed to protect their technology. But the combination of claims didn’t seem to line up with any known chemistries, leaving experts speculating and, in many cases, doubting Donut Lab’s claims. “All the parameters are contradictory,” said Yang Hongxin, chairman and CEO of the Chinese battery giant Svolt Energy, in remarks to news outlets in January. For example, there’s often a trade-off between high energy density, which requires thicker electrodes that can store more energy, and fast charging, which requires ions to move quickly through cells. High-performance batteries are also expected to be costly, but Donut Lab claims its technology will be cheaper than lithium-ion technology.  In a new video released last week, Donut Lab cofounder and CEO Marko Lehtimäki announced the company would be releasing a video series, called “I Donut Believe,” that would provide evidence for their claims. As a header on the accompanying website reads: “Fair enough. Here you go.” When the website went up last week, it included a countdown timer to Monday February 23, when the company released results from its first third-party testing: a fast charging test. The test showed that a single cell could charge from 0% to 80% capacity in about four and a half minutes—incredibly quick and quite impressive results. (One potential caveat to note is that the cells heated up quite a bit, so thermal management could be important in designing vehicles that use these batteries.) Even as we see the first technical test results, I’m still left with a lot of questions. How many cycles could this battery do at this charging speed? Can this same cell meet the company’s other performance claims? (I’ve reached out to Donut Lab several times over the past month, both to the company’s press email and to leadership on LinkedIn, but I haven’t gotten a response yet.) The company has certainly drummed up a lot of interest and attention with its rollout, and the theatrics aren’t over yet. There’s another countdown timer on Donut Lab’s site, which ends on Monday, March 2. I’m the first one to get excited about a new battery technology. But there’s a sentiment I’ve seen pop up a lot recently online, and

Eons ago, in 2012, I had a weird experience. My iPhone suddenly shut down. When I restarted it, I found it was totally reset—clean, like a new device. This was the early days of iOS, so I wasn’t too concerned until I went to connect it to my computer to restore it from a backup. But when I flipped open the lid of my laptop, it too was mid-restart. And then, suddenly, the screen went gray. It was being remotely wiped. I turned on my iPad. It, too, had been wiped. I was being hacked.  Frantically, I shut down all my devices, unplugged everything connected to the internet in my house, turned off my router, and went next door to use my neighbors’ computer and find out what was going on. Deepening my panic, I realized hackers had also gained control of, and nuked, my Google account. Worse, they were in control of my Twitter, which they were gleefully using to spew all sorts of vile comments. It was nasty.  You have to remember, this was before all of us lived with a constant rain of text messages and emails designed to elicit the information necessary to pull something like this off. These crooks hadn’t brute-forced their way in, or used any sort of sophisticated techniques to gain access to my accounts. Instead, they had relied on publicly available information, and a fake credit card number, to socially engineer their way into my Amazon account, where they looked up the last four digits of my real credit card number. Then they used that information to get into Apple. And because that account was linked to my Gmail, and that to my Twitter, it gave them the keys to everything. But what really troubled me was what I learned as I followed up on my hack over the ensuing weeks and months: This kind of thing was, while still novel, becoming more common. Some version of what happened to me had happened to lots of other people. The kids who were responsible—it was a couple of kids—weren’t criminal masterminds. They had just found a gap, a place where a technology was now commonplace but its risks and exploitable surface areas weren’t yet fully understood. I just happened to have all my stuff in the gap. Today that gap might feature a crypto wallet or a deepfake of a loved one’s voice. (Or both.) Crime changes. The goals stay the same—pursuit of value, pursuit of power—but new technologies create new vulnerabilities, new tactics, and new ways for perpetrators to evade discovery or capture. And the law necessarily lags behind. Relying not on innovation but on precedent, it is intentionally backward-looking and slow. That plodding consideration used to be how we protected our shared democratic society, how we protected each other from each other. But those same new technologies that have allowed crime to outpace law have also reenergized law enforcement and government—offering new ways to root out crime, to gather evidence, to surveil people. Think, for example, of how cold-case investigators tracked down the Golden State Killer years after his murders, using DNA samples and genealogy databases—launching a new era of DNA-powered investigations.  Technology has long made crime and its prosecution a game of cat and mouse. It sometimes calls into question the nature of crime itself. Unregulated behaviors, facilitated by technology, can exist in murky zones of dubious legality. (Until TikTok announced its new ownership structure, Apple and Google were both technically breaking the law by allowing the app to stay on their platforms, under the provisions of the Protecting Americans from Foreign Adversary Controlled Applications Act. Ah! Well. Nevertheless.) That tension is the key to our March/April issue. Thanks to technologies like cryptocurrency and off-the-shelf autonomous autopilots, there’s never been a better time to do crime. Thanks to pervasive surveillance and digital infrastructure, there’s never been a better time to fight it—sometimes at the expense of what we used to think of as fundamental civil rights.  I never pressed charges against the kids who hacked me. The biggest consequence of the hack was that Apple set up two-factor authentication in the following months, which felt like a win. Now I’m not sure anyone expects their personal data to be secure in any meaningful way. I’m certain, though, that somewhere on the net, a new generation of kids is coming up with another novel crime. 

The only reality show that matters The Real Housewives of Salt Lake City is one of the best shows on television right now. Not one of the best reality TV shows, but one of the best TV shows, period. Chronicling a shifting group of wealthy women in and around Salt Lake, the show has featured a convicted felon whom federal agents came looking for while cameras were rolling, a church leader married to her step-grandfather, and a single mom in an exhausting on-again, off-again relationship with an Osmond. In one season, there was an ongoing argument between two cast members after one told the other that she “smelled like hospital.” Later, one woman was secretly running an anonymous gossip Instagram about her fellow housewives. We can debate the “reality” of reality television, and it’s certainly true that these characters and scenarios are far-fetched. But every single person is dealing with something relatable—difficult marriages, failing businesses, strained relationships with children, addiction. It’s entertainment, and high camp, but I find that I still have a lot of empathy for these people. The last good place(s) on Facebook Facebook sucks. That’s not controversial to say, right? But there is one reason I still have a Facebook account: my neighborhood Buy Nothing group. The spirit of community and camaraderie is alive and well there—and probably in yours, too. A non-exhaustive list of things I have given away: empty candle jars, a bookcase, used lightbulbs, unopened toiletries, bubble wrap. I’ve scored a few good things as well: a gorgeous antique dresser that I refinished, some over-the-door hooks, and brand-new jeans. It makes me happy to know that stuff that would’ve otherwise ended up in a landfill is bringing one of my neighbors joy. Going analog I used to wear an Apple Watch a lot. I’m a pretty active person, and I liked tracking my workouts and my steps. But after I’d had it for a while, my watch started dying in the middle of a 30-minute run; it became useless to me, and I gave it up completely. Guess what? I’m happier. I feel more present when I’m not checking how much time is left in a yoga class or reading texts during a long run. The amount of data it gathered about me was also stressing me out, and it wasn’t useful. And I don’t need a wearable to tell me how poorly I slept! Trust me, I already know.

The boom of a calving glacier. The crackling rumble of a wildfire. The roar of a surging storm front. They’re the noises of the living Earth, music of this one particular sphere and clues to the true nature of these dramatic events. But as loud as all these things are, they emit even more acoustic energy below the threshold of human hearing, at frequencies of 20 hertz or lower. These “infrasounds” have such long wavelengths that they can travel around the globe as churning emanations of distant events. But humans have never been able to hear them. Until now, that is. Everyday Infrasound in an Uncertain World, a new album by the musician and artist Brian House, condenses 24 hours of these rumbles into 24 minutes of the most basic of bass lines, putting a new spin on the idea of ambient music. Sound, even infrasound, is really just variations in air pressure. So House built a set of three “macrophones,” tubes that funnel air into a barometer capable of taking readings 100 times a second. From the quiet woods of western Massachusetts, House can pick up what the planet is laying down. Then he speeds the recording up by a factor of 60 so that it’s audible to the wee ears of humans. “I am really interested in the layers of perception that we can’t access,” he says. “It’s not only low sound, but it’s also distant sound. That kind of blew my mind.” House’s album is art, but scientists made it possible. Barometers picked up the 1883 eruption of the South Pacific volcano Krakatoa as far away as London. And today, a global network of infrasound sensors helps enforce the nuclear test ban treaty. A few infrasound experts—like Leif Karlstrom, a volcanologist at the University of Oregon who uses infrasound to study Mount Kilauea in Hawaii—helped House set up his music-gathering array and better understand what he was hearing. “He’s highlighting interesting phenomena,” Karlstrom says, even though it’s impossible to tell exactly what is making each specific sound.  So how’s the actual music? It’s 24 minutes of an otherworldly chorus, alternating between low grumbling vibrations and soft ghostlike whispers. A high-pitched whistle? Could be a train, House says. An intense low-octave rattle? Maybe a distant thunderstorm or a shifting ocean current. “For me, it’s about the mystery of it,” he says. “I hope that’s a little bit unsettling.” But it also might connect someone listening to a wider—and deeper—world.  Monique Brouillette is a freelance writer based in Cambridge, Massachusetts.