Uncategorized

As AI systems enter production, reliability and governance can’t depend on wishful thinking. Here’s how observability turns large language models (LLMs) into auditable, trustworthy enterprise systems. Why observability secures the future of enterprise AI The enterprise race to deploy LLM systems mirrors the early days of cloud adoption. Executives love the promise; compliance demands accountability; engineers just want a paved road. Yet, beneath the excitement, most leaders admit they can’t trace how AI decisions are made, whether they helped the business, or if they broke any rule. Take one Fortune 100 bank that deployed an LLM to classify loan applications. Benchmark accuracy looked stellar. Yet, 6 months later, auditors found that 18% of critical cases were misrouted, without a single alert or trace. The root cause wasn’t bias or bad data. It was invisible. No observability, no accountability. If you can’t observe it, you can’t trust it. And unobserved AI will fail in silence. Visibility isn’t a luxury; it’s the foundation of trust. Without it, AI becomes ungovernable. Start with outcomes, not models Most corporate AI projects begin with tech leaders choosing a model and, later, defining success metrics. That’s backward. Flip the order: Define the outcome first. What’s the measurable business goal? Deflect 15 % of billing calls Reduce document review time by 60 % Cut case-handling time by two minutes Design telemetry around that outcome, not around “accuracy” or “BLEU score.” Select prompts, retrieval methods and models that demonstrably move those KPIs. At one global insurer, for instance, reframing success as “minutes saved per claim” instead of “model precision” turned an isolated pilot into a company-wide roadmap. A 3-layer telemetry model for LLM observability Just like microservices rely on logs, metrics and traces, AI systems need a structured observability stack: a) Prompts and context: What went in Log every prompt template, variable and retrieved document. Record model ID, version, latency and token counts (your leading cost indicators). Maintain an auditable redaction log showing what data was masked, when and by which rule. b) Policies and controls: The guardrails Capture safety-filter outcomes (toxicity, PII), citation presence and rule triggers. Store policy reasons and risk tier for each deployment. Link outputs back to the governing model card for transparency. c) Outcomes and feedback: Did it work? Gather human ratings and edit distances from accepted answers. Track downstream business events, case closed, document approved, issue resolved. Measure the KPI deltas, call time, backlog, reopen rate. All three layers connect through a common trace ID, enabling any decision to be replayed, audited or improved. Diagram © SaiKrishna Koorapati (2025). Created specifically for this article; licensed to VentureBeat for publication. Apply SRE discipline: SLOs and error budgets for AI Service reliability engineering (SRE) transformed software operations; now it’s AI’s turn. Define three “golden signals” for every critical workflow: Signal Target SLO When breached Factuality ≥ 95 % verified against source of record Fallback to verified template Safety ≥ 99.9 % pass toxicity/PII filters Quarantine and human review Usefulness ≥ 80 % accepted on first pass Retrain or rollback prompt/model If hallucinations or refusals exceed budget, the system auto-routes to safer prompts or human review just like rerouting traffic during a service outage. This isn’t bureaucracy; it’s reliability applied to reasoning. Build the thin observability layer in two agile sprints You don’t need a six-month roadmap, just focus and two short sprints. Sprint 1 (weeks 1-3): Foundations Version-controlled prompt registry Redaction middleware tied to policy Request/response logging with trace IDs Basic evaluations (PII checks, citation presence) Simple human-in-the-loop (HITL) UI Sprint 2 (weeks 4-6): Guardrails and KPIs Offline test sets (100–300 real examples) Policy gates for factuality and safety Lightweight dashboard tracking SLOs and cost Automated token and latency tracker In 6 weeks, you’ll have the thin layer that answers 90% of governance and product questions. Make evaluations continuous (and boring) Evaluations shouldn’t be heroic one-offs; they should be routine. Curate test sets from real cases; refresh 10–20 % monthly. Define clear acceptance criteria shared by product and risk teams. Run the suite on every prompt/model/policy change and weekly for drift checks. Publish one unified scorecard each week covering factuality, safety, usefulness and cost. When evals are part of CI/CD, they stop being compliance theater and become operational pulse checks. Apply human oversight where it matters Full automation is neither realistic nor responsible. High-risk or ambiguous cases should escalate to human review. Route low-confidence or policy-flagged responses to experts. Capture every edit and reason as training data and audit evidence. Feed reviewer feedback back into prompts and policies for continuous improvement. At one health-tech firm, this approach cut false positives by 22 % and produced a retrainable, compliance-ready dataset in weeks. Cost control through design, not hope LLM costs grow non-linearly. Budgets won’t save you architecture will. Structure prompts so deterministic sections run before generative ones. Compress and rerank context instead of dumping entire documents. Cache frequent queries and memoize tool outputs with TTL. Track latency, throughput and token use per feature. When observability covers tokens and latency, cost becomes a controlled variable, not a surprise. The 90-day playbook Within 3 months of adopting observable AI principles, enterprises should see: 1–2 production AI assists with HITL for edge cases Automated evaluation suite for pre-deploy and nightly runs Weekly scorecard shared across SRE, product and risk Audit-ready traces linking prompts, policies and outcomes At a Fortune 100 client, this structure reduced incident time by 40 % and aligned product and compliance roadmaps. Scaling trust through observability Observable AI is how you turn AI from experiment to infrastructure. With clear telemetry, SLOs and human feedback loops: Executives gain evidence-backed confidence. Compliance teams get replayable audit chains. Engineers iterate faster and ship safely. Customers experience reliable, explainable AI. Observability isn’t an add-on layer, it’s the foundation for trust at scale. SaiKrishna Koorapati is a software engineering leader. Read more from our guest writers. Or, consider submitting a post of your own! See our guidelines here.

Why do current audio AI models often perform worse when they generate longer reasoning instead of grounding their decisions in the actual sound. StepFun research team releases Step-Audio-R1, a new audio LLM designed for test time compute scaling, address this failure mode by showing that the accuracy drop with chain of thought is not an audio limitation but a training and modality grounding problem? https://arxiv.org/pdf/2511.15848 The Core Problem, Audio Models Reason over Text Surrogates Most current audio models inherit their reasoning behavior from text training. They learn to reason as if they read transcripts, not as if they listen. The StepFun team calls this Textual Surrogate Reasoning. The model uses imagined words and descriptions instead of acoustic cues such as pitch contour, rhythm, timbre or background noise patterns. This mismatch explains why longer chain of thought often hurts performance in audio. The model spends more tokens elaborating wrong or modality irrelevant assumptions. Step-Audio-R1 attacks this by forcing the model to justify answers using acoustic evidence. The training pipeline is organized around Modality Grounded Reasoning Distillation, MGRD, which selects and distills reasoning traces that explicitly reference audio features. Architecture The architecture stays close to the previous Step Audio systems: A Qwen2 based audio encoder processes raw waveforms at 25 Hz. An audio adaptor downsamples the encoder output by a factor of 2, to 12.5 Hz, and aligns frames to the language token stream. A Qwen2.5 32B decoder consumes the audio features and generates text. The decoder always produces an explicit reasoning block inside <think> and </think> tags, followed by the final answer. This separation lets training objectives shape the structure and content of reasoning without losing focus on task accuracy. The model is released as a 33B parameter audio text to text model on Hugging Face under Apache 2.0. https://arxiv.org/pdf/2511.15848 Training Pipeline, from Cold Start to Audio Grounded RL The pipeline has a supervised cold start stage and a reinforcement learning stage that both mix text and audio tasks. Cold start uses about 5 million examples, covering 1 billion tokens of text only data and 4 billion tokens from audio paired data. Audio tasks include automatic speech recognition, paralinguistic understanding and audio question text answer style dialogs. A fraction of the audio data carries audio chain of thought traces generated by an earlier model. Text data covers multi turn dialog, knowledge question answering, math and code reasoning. All samples share a format where reasoning is wrapped in <think> tags, even when the reasoning block is initially empty. Supervised learning trains Step-Audio-R1 to follow this format and to generate useful reasoning for both audio and text. This gives a baseline chain of thought behavior, but it is still biased toward text based reasoning. Modality Grounded Reasoning Distillation MGRD MGRD is applied in several iterations. For each round, the research team samples audio questions where the label depends on real acoustic properties. For example, questions about speaker emotion, background events in sound scenes or musical structure. The current model produces multiple reasoning and answer candidates per question. A filter keeps only chains that meet three constraints: They reference acoustic cues, not just textual descriptions or imagined transcripts. They are logically coherent as short step by step explanations. Their final answers are correct according to labels or programmatic checks. These accepted traces form a distilled audio chain of thought dataset. The model is fine tuned on this dataset together with the original text reasoning data. This is followed by Reinforcement Learning with Verified Rewards, RLVR. For text questions, rewards are based on answer correctness. For audio questions, the reward mixes answer correctness and reasoning format, with a typical weighting of 0.8 for accuracy and 0.2 for reasoning. Training uses PPO with about 16 responses sampled per prompt and supports sequences up to around 10 240 tokens to allow long deliberation. https://arxiv.org/pdf/2511.15848 Benchmarks, closing the gap to Gemini 3 Pro On a combined speech to text benchmark suite that includes Big Bench Audio, Spoken MQA, MMSU, MMAU and Wild Speech, Step-Audio-R1 reaches an average score of about 83.6 percent. Gemini 2.5 Pro reports about 81.5 percent and Gemini 3 Pro reaches about 85.1 percent. On Big Bench Audio alone, Step-Audio-R1 reaches about 98.7 percent, which is higher than both Gemini versions. For speech to speech reasoning, the Step-Audio-R1 Realtime variant adopts listen while thinking and think while speaking style streaming. On Big Bench Audio speech to speech, it reaches about 96.1 percent reasoning accuracy with first packet latency around 0.92 seconds. This score surpasses GPT based realtime baselines and Gemini 2.5 Flash style native audio dialogs while keeping sub second interaction. https://arxiv.org/pdf/2511.15848 Ablations, what matters for audio reasoning The ablation section provides several design signals for engineers: A reasoning format reward is necessary. Without it, reinforcement learning tends to shorten or remove chain of thought, which lowers audio benchmark scores. RL data should target medium difficulty problems. Selecting questions where pass at 8 lies in a middle band gives more stable rewards and maintains long reasoning. Scaling RL audio data without such selection does not help. Quality of prompts and labels matters more than raw size. The researchers also describe a self cognition correction pipeline that reduces the frequency of answers such as ‘I can only read text and cannot hear audio’ in a model that is trained to process sound. This uses Direct Preference Optimization on curated preference pairs where correct behavior is to acknowledge and use audio input. Key Takeaways Step-Audio-R1 is one of the first audio language model that turns longer chain of thought into a consistent accuracy gain for audio tasks, solving the inverted scaling failure seen in previous audio LLMs. The model explicitly targets Textual Surrogate Reasoning by using Modality Grounded Reasoning Distillation, which filters and distills only those reasoning traces that rely on acoustic cues such as pitch, timbre and rhythm instead of imagined transcripts. Architecturally, Step-Audio-R1 combines a Qwen2 based audio encoder with an adaptor and a Qwen2.5 32B decoder that always generates <think> reasoning segments before

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. What we still don’t know about weight-loss drugs Weight-loss drugs have been back in the news this week. First, we heard that Eli Lilly, the company behind Mounjaro and Zepbound, became the first healthcare company in the world to achieve a trillion-dollar valuation. But we also learned that, disappointingly, GLP-1 drugs don’t seem to help people with Alzheimer’s disease. And that people who stop taking the drugs when they become pregnant can experience potentially dangerous levels of weight gain. On top of that, some researchers worry that people are using the drugs postpartum to lose pregnancy weight without understanding potential risks. All of this news should serve as a reminder that there’s a lot we still don’t know about these drugs. So let’s look at the enduring questions surrounding GLP-1 agonist drugs. —Jessica Hamzelou This article first appeared in The Checkup, MIT Technology Review’s weekly biotech newsletter. To receive it in your inbox every Thursday, and read articles like this first, sign up here. If you’re interested in weight loss drugs and how they affect us, take a look at: + GLP-1 agonists like Wegovy, Ozempic, and Mounjaro might benefit heart and brain health—but research suggests they might also cause pregnancy complications and harm some users. Read the full story. + We’ve never understood how hunger works. That might be about to change. Read the full story. + Weight-loss injections have taken over the internet. But what does this mean for people IRL? + This vibrating weight-loss pill seems to work—in pigs. Read the full story. What we know about how AI is affecting the economy There’s a lot at stake when it comes to understanding how AI is changing the economy right now. Should we be pessimistic? Optimistic? Or is the situation too nuanced for that? Hopefully, we can point you towards some answers. Mat Honan, our editor in chief, will hold a special subscriber-only Roundtables conversation with our editor at large David Rotman, and Richard Waters, Financial Times columnist, exploring what’s happening across different markets. Register here to join us at 1pm ET on Tuesday December 9. The event is part of the Financial Times and MIT Technology Review “The State of AI” partnership, exploring the global impact of artificial intelligence. Over the past month, we’ve been running discussions between our journalists—sign up here to receive future editions every Monday. The must-reads I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology. 1 Tech billionaires are gearing up to fight AI regulation By amassing multi-million dollar war chests ahead of the 2026 US midterm elections. (WSJ $)+ Donald Trump’s “Manhattan Project” for AI is certainly ambitious. (The Information $) 2 The EU wants to hold social media platforms liable for financial scamsNew rules will force tech firms to compensate banks if they fail to remove reported scams. (Politico) 3 China is worried about a humanoid robot bubbleBecause more than 150 companies there are building very similar machines. (Bloomberg $)+ It could learn some lessons from the current AI bubble. (CNN)+ Why the humanoid workforce is running late. (MIT Technology Review) 4 A Myanmar scam compound was blown upBut its residents will simply find new bases for their operations. (NYT $)+ Experts suspect the destruction may have been for show. (Wired $)+ Inside a romance scam compound—and how people get tricked into being there. (MIT Technology Review) 5 Navies across the world are investing in submarine drones They cost a fraction of what it takes to run a traditional manned sub. (The Guardian)+ How underwater drones could shape a potential Taiwan-China conflict. (MIT Technology Review) 6 What to expect from China’s seemingly unstoppable innovation driveIts extremely permissive regulators play a big role. (Economist $)+ Is China about to win the AI race? (MIT Technology Review) 7 The UK is waging a war on VPNsGood luck trying to persuade people to stop using them. (The Verge) 8 We’re learning more about Jeff Bezos’ mysterious clock projectHe’s backed the Clock of the Long Now for years—and construction is amping up. (FT $)+ How aging clocks can help us understand why we age—and if we can reverse it. (MIT Technology Review) 9 Have we finally seen the first hints of dark matter?These researchers seem to think so. (New Scientist $) 10 A helpful robot is helping archaeologists reconstruct PompeiiReassembling ancient frescos is fiddly and time-consuming, but less so if you’re a dextrous machine. (Reuters) Quote of the day “We do fail… a lot.” —Defense company Anduril explains its move-fast-and-break-things ethos to the Wall Street Journal in response to reports its systems have been marred by issues in Ukraine. One more thing How to build a better AI benchmark It’s not easy being one of Silicon Valley’s favorite benchmarks. SWE-Bench (pronounced “swee bench”) launched in November 2024 as a way to evaluate an AI model’s coding skill. It has since quickly become one of the most popular tests in AI. A SWE-Bench score has become a mainstay of major model releases from OpenAI, Anthropic, and Google—and outside of foundation models, the fine-tuners at AI firms are in constant competition to see who can rise above the pack. Despite all the fervor, this isn’t exactly a truthful assessment of which model is “better.” Entrants have begun to game the system—which is pushing many others to wonder whether there’s a better way to actually measure AI achievement. Read the full story. —Russell Brandom We can still have nice things A place for comfort, fun and distraction to brighten up your day. (Got any ideas? Drop me a line or skeet ’em at me.) + Aww, these sharks appear to be playing with pool toys.+ Strange things are happening over on Easter Island (even weirder than you can imagine) + Very cool—archaeologists have uncovered a Roman tomb that’s been sealed shut for 1,700 years.+ This Japanese mass media collage is making my

Agent memory remains a problem that enterprises want to fix, as agents forget some instructions or conversations the longer they run.  Anthropic believes it has solved this issue for its Claude Agent SDK, developing a two-fold solution that allows an agent to work across different context windows. “The core challenge of long-running agents is that they must work in discrete sessions, and each new session begins with no memory of what came before,” Anthropic wrote in a blog post. “Because context windows are limited, and because most complex projects cannot be completed within a single window, agents need a way to bridge the gap between coding sessions.” Anthropic engineers proposed a two-fold approach for its Agent SDK: An initializer agent to set up the environment, and a coding agent to make incremental progress in each session and leave artifacts for the next.   The agent memory problem Since agents are built on foundation models, they remain constrained by the limited, although continually growing, context windows. For long-running agents, this could create a larger problem, leading the agent to forget instructions and behave abnormally while performing a task. Enhancing agent memory becomes essential for consistent, business-safe performance.  Several methods emerged over the past year, all attempting to bridge the gap between context windows and agent memory. LangChain’s LangMem SDK, Memobase and OpenAI’s Swarm are examples of companies offering memory solutions. Research on agentic memory has also exploded recently, with proposed frameworks like Memp and the Nested Learning Paradigm from Google offering new alternatives to enhance memory.  Many of the current memory frameworks are open source and can ideally adapt to different large language models (LLMs) powering agents. Anthropic’s approach improves its Claude Agent SDK.  How it works Anthropic identified that even though the Claude Agent SDK had context management capabilities and “should be possible for an agent to continue to do useful work for an arbitrarily long time,” it was not sufficient. The company said in its blog post that a model like Opus 4.5 running the Claude Agent SDK can “fall short of building a production-quality web app if it’s only given a high-level prompt, such as ‘build a clone of claude.ai.’”  The failures manifested in two patterns, Anthropic said. First, the agent tried to do too much, causing the model to run out of context in the middle. The agent then has to guess what happened and cannot pass clear instructions to the next agent. The second failure occurs later on, after some features have already been built. The agent sees progress has been made and just declares the job done.  Anthropic researchers broke down the solution: Setting up an initial environment to lay the foundation for features and prompting each agent to make incremental progress towards a goal, while still leaving a clean slate at the end.  This is where the two-part solution of Anthropic’s agent comes in. The initializer agent sets up the environment, logging what agents have done and which files have been added. The coding agent will then ask models to make incremental progress and leave structured updates.  “Inspiration for these practices came from knowing what effective software engineers do every day,” Anthropic said.  The researchers said they added testing tools to the coding agent, improving its ability to identify and fix bugs that weren’t obvious from the code alone.  Future research Anthropic noted that its approach is “one possible set of solutions in a long-running agent harness.” However, this is just the beginning stage of what could become a wider research area for many in the AI space.  The company said its experiments to boost long-term memory for agents haven’t shown whether a single general-purpose coding agent works best across contexts or a multi-agent structure.  Its demo also focused on full-stack web app development, so other experiments should focus on generalizing the results across different tasks. “It’s likely that some or all of these lessons can be applied to the types of long-running agentic tasks required in, for example, scientific research or financial modeling,” Anthropic said.