Uncategorized

LLM applications fail in ways traditional software does not. The same prompt can produce different outputs. A retrieval step can return the wrong document while every HTTP status reads 200. An agent can loop through fourteen tool calls, burn thousands of tokens, and deliver a confidently wrong answer. Standard application performance monitoring (APM) alone does not capture this semantic behavior — prompt and output quality, retrieval relevance, or agent-level reasoning traces. This is the gap LLM observability and evaluation platforms fill. They record every span of an LLM pipeline — prompts, completions, retrievals, tool calls, token counts, latencies, and costs — and then score outputs for quality using automated evaluators. In 2026, this category has moved from optional tooling to core infrastructure for any team running AI in production. The market data reflects the shift. The Business Research Company sizes the LLM observability platform market at $2.69 billion in 2026, up from $1.97 billion in 2025, and projects $9.26 billion by 2030 at a 36.2% forecast CAGR. Gartner predicts that by 2028, LLM observability investments will account for 50% of GenAI deployments, up from 15% in early 2026. LangChain’s State of Agent Engineering survey of 1,300+ professionals found that 57% of respondents now run agents in production. Nearly 89% have implemented observability for their agents. Evaluation lags behind: 52.4% run offline evaluations, 37.3% run online evaluations, and 29.5% report no evaluation at all. Quality was cited by 32% as the top barrier to production deployment. This article compares the leading platforms across three axes: tracing depth, evaluation capability, and production monitoring. Figures were checked against primary sources (company documentation, press pages, and announcements) as of August 2026; where only secondary reporting exists, it is linked and identified as such. Rankings and “best for” judgments are editorial assessments, not measured benchmarks. How the Category is Structured in 2026 The market has split into four camps, and understanding the split matters more than any individual feature list. AI-native observability platforms: Langfuse, LangSmith, Braintrust, Arize, Opik — treat the LLM trace as the primary object. They capture nested spans across agents, retrievers, and tools, and attach evaluation scores to production traffic. Open-source and source-available evaluation libraries and platforms: Arize Phoenix, DeepEval (Confident AI), MLflow, RAGAS — focus on scoring outputs: faithfulness, hallucination, answer relevance, and task completion, often via LLM-as-a-judge. AI gateways: Helicone, Portkey, LiteLLM — sit as a proxy between the application and model providers. They add logging, caching, cost tracking, and routing with minimal code changes. APM extensions: Datadog LLM Observability, New Relic, Dynatrace — bolt LLM tracing onto existing infrastructure monitoring so AI signals correlate with CPU, memory, and network metrics. One standard now connects all four camps. The OpenTelemetry GenAI semantic conventions define vendor-neutral gen_ai.* span attributes for model calls, token usage, agent steps, and tool executions. OpenTelemetry, a CNCF project, maintains these conventions, which are adopted by platforms including Google Cloud, AWS, Azure, and Datadog. The conventions now live in a dedicated repository, with the GenAI registry under active development as of August 2026. Coding agents are converging on the standard too: GitHub Copilot’s agent telemetry exposes gen_ai.* span trees, Claude Code provides opt-in OpenTelemetry tracing, and Codex includes native OpenTelemetry export support. Instrumenting once against gen_ai.* improves backend portability and reduces vendor-specific instrumentation, even if implementations still differ. Buyers in 2026 should treat OTel compatibility as a hard requirement, not a nice-to-have. The Three Axes: Tracing, Evals, and Production Monitoring Because vendors use these terms loosely, precise definitions help before comparing platforms: Tracing is the record of everything an LLM application did. A trace contains nested spans: the user input, each retrieval call, each model invocation with its exact prompt and parameters, each tool execution, and the final output. Depth matters because agent traces are deeply nested with heavy payloads — a single conversation can generate megabytes of data across dozens of runs and tool calls. Non-determinism makes tracing non-negotiable: the same prompt produces different outputs, so an issue cannot be reproduced without capturing the exact input, model parameters, and temperature at call time. Evals answer the question tracing cannot: was the output any good? Offline evals score curated datasets before deployment, catching regressions when a prompt, model, or retrieval index changes. Online evals score live production traffic, typically via LLM-as-a-judge, sampling traces and grading them for faithfulness, relevance, toxicity, or task completion. The hardest failures are outputs that are technically valid but wrong for the domain — a hallucinated policy, a drifting tone, a retrieval miss that produces a confident but incorrect answer. Traditional latency, error-rate, and availability metrics do not detect these semantic quality failures. Production monitoring closes the loop: dashboards, cost attribution per model and user, latency percentiles, drift detection across prompts and use cases, and alerting when quality scores fall. The best platforms feed production traces back into eval datasets, so every real-world failure becomes a future regression test. A platform can be strong on one axis and weak on another. Gateways excel at monitoring but skip deep tracing. Eval libraries score outputs but do not watch production. The platforms below are ranked on how completely they cover all three. 1. Langfuse (ClickHouse) Langfuse describes itself as the most widely adopted LLM engineering platform, and its open-source adoption numbers back a strong claim. Tracing: Langfuse captures nested traces for LLM calls, retrieval, embedding, and agent actions through OpenTelemetry, LangChain, OpenAI SDK, and LiteLLM integrations. Its signature nested trace view collapses a multi-step RAG or agent run into a stepable tree with per-span latencies and token counts. An observations-centric data model shipped in March 2026, delivering 10x+ dashboard performance gains and laying the groundwork for Langfuse v4, which the company says runs up to 165x faster. Evals: The platform supports LLM-as-a-judge evaluators, human annotation queues, custom scores, and dataset-based regression testing that runs in CI via GitHub Actions. Evaluator templates cover hallucination, toxicity, and relevance. Production monitoring: Cost breakdowns by model, user, or session, plus session replays for conversational agents. Deployment: MIT-licensed core,

Mistral AI has released Shieldstral 1.0 3B, an open-weights, policy-adaptive multimodal safety classifier that treats content moderation as a single yes/no question rather than a fixed taxonomy of harm categories. Most guardrail models bake their category list into the weights, so re-targeting one to a new deployment context means retraining — and the same content can be acceptable on a cybersecurity research tool while being harmful on a mental-health platform. Shieldstral inverts that: operators write the policy as a plain-language question at inference time, and the model returns a calibrated safety score from a single forward pass. Built on Ministral-3-3B-Base-2512 with a native Pixtral vision encoder and released under Apache 2.0, it reports 84.9% average F1 on text safety — matching GPT-OSS-Safeguard-20B — and 83.8% on multimodal safety, ahead of every baseline Mistral evaluated. Is it deployable? Yes, and locally. Shieldstral-1.0-3B fits in 16GB of VRAM in BF16, runs on a single GPU, and is licensed Apache 2.0 for commercial and non-commercial use. Serving paths are already in place: vLLM (≥0.26.0, recommended), llama.cpp via GGUF conversion with Q8_0/Q5_K_M/Q4_K_M quantization, SGLang, and Transformers — with fine-tuning supported through Axolotl. The classifier emits one token, so latency and cost sit far below reasoning-based guards like GPT-OSS-Safeguard-20B. Which level of company: the 16GB footprint puts it within reach of seed-stage AI product teams that cannot justify a moderation vendor contract, while the open license and self-hosting story suit mid-market and enterprise teams that need guardrails inside a VPC or on-prem for data-residency and audit reasons. Multi-tenant SaaS vendors get a specific win — one checkpoint can enforce a different policy per customer. Industries: consumer social and UGC platforms, ed-tech and child-safety surfaces, healthcare and mental-health apps, fintech and insurance support automation, gaming and voice chat, marketplaces and ad/creative review, and public sector deployments with sovereignty requirements. Applications: user-prompt moderation, model-response moderation, refusal classification, image-plus-caption review for ads and memes, training-data and RAG-corpus curation, output gating in agentic pipelines, and per-tenant policy enforcement. Because the output is a continuous score rather than a label, teams can tune the threshold per surface or route borderline scores to human review instead of hard-blocking. Moderation as a binary question Shieldstral reduces moderation to one yes/no question. A fixed system message establishes the task; the user message carries three fields: <Instruct> (evaluation context and strictness), <Query> (the policy, phrased as a single yes/no question), and <Document> (a prompt, a response, a prompt–response pair, or an image with optional text). At inference the model unembeds only toward the yes and no token IDs and softmax-normalizes them into a continuous score, thresholded at τ=0.5. That collapses prompt classification, response moderation, refusal detection, and toxicity detection into one problem — and it means the policy lives entirely in the prompt. Mistral’s guidance is one policy per call; for a broad safe/unsafe verdict, list the categories in <Instruct> and ask a single wide <Query>. The data recipe The claimed advantage comes from data, not scale: roughly 54.1M samples — 45.2M open-source text, 4.4M synthetic contrastive text, 4.5M multimodal. A template-based unification layer converts every dataset into the same instruction–query–document format via per-dataset processors, with randomized phrasings and calibrated strictness (strict for adversarial jailbreaks, lenient for response-quality data). The more interesting piece is contrastive generation. An LLM rewrites safe text into an unsafe variant that violates a target category but deliberately not its sibling, producing a positive and a hard negative over identical content in one call. That teaches the model which policy is violated rather than a coarse safe/unsafe split. Image data — which cannot be synthesized the way text can — is supplemented with general-purpose image datasets as negatives, query mutation across a 14-subcategory visual taxonomy, and vision–language reranker filtering. Training is LoRA fine-tuning followed by a three-way SLERP merge: 0.6 public+generated, 0.3 public-only, 0.1 Ministral-3B-Instruct. Results On text safety, Shieldstral reports 84.9% average F1, tying GPT-OSS-Safeguard-20B (84.9%) as the smallest model in the comparison, with wins on ToxicChat (84.1), HarmBench (99.4), and Aegis v2 response (87.2). On multimodal safety it reports 83.8% overall versus 77.6% for OmniGuard-7B, leading VLGuard (97.7) and UnsafeBench (81.8); LlavaGuard-7B still leads its namesake benchmark at 81.4. On the adaptability benchmark — built on a deliberately divergent taxonomy of 12 super classes, 26 subcategories, and 52 leaf categories with 90 fixed queries, where no leaf maps one-to-one to training — Shieldstral scores 91.3% F1, behind GPT-OSS-Safeguard-20B (94.1%) and Nemotron-3.5-Safety-4B (91.8%), but without generating a reasoning trace. Refusal detection lands at 91.5% overall against 93.7% for GPT-OSS-Safeguard-20B. Where it is weaker: multilingual prompt classification lags on Arabic and Indonesian and on RTP-LX prompts (70.3 vs 86.1 for Nemotron-3.5-Safety-4B). Mistral also flags reduced reliability on adversarial or obfuscated inputs and very long documents. Trained context is 32k tokens across 12 languages. Key Takeaways 3B Apache 2.0 multimodal guardrail; policy is a plain-language question at inference time, no retraining. 84.9% text F1 ties a 20B model; 83.8% multimodal F1 is best-in-class among evaluated baselines. 54.1M samples with sibling-contrastive rewrites is the actual mechanism behind policy generalization. Single forward pass, single token out, continuous score at τ=0.5 — cheap enough for real-time gating. Weak spots: low-resource languages, obfuscated inputs, long documents. Check out the Paper, Model on Hugging Face, and Technical Details. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Mistral AI Releases Shieldstral 1.0 3B: An Open-Weights Policy-Adaptive Multimodal Safety Classifier Matching Models 7× Its Size appeared first on MarkTechPost.

arXiv:2608.05510v1 Announce Type: new Abstract: Dialectal variation remains a major challenge for multilingual language models. Perturbation-based continued pre-training (CPT) has emerged as a promising approach to improving robustness, yet existing work largely evaluates individual perturbation strategies in isolation and provides limited insight into why they work. We present a systematic study of perturbation-based CPT for multilingual dialect robustness in LLMs, comparing six training conditions across nine German, Italian, and Arabic dialect tasks. Perturbation-based CPT, especially character-noised CPT, consistently improves zero-shot dialect robustness while largely preserving standard variety performance. More importantly, we show that methods with similar downstream performance induce distinct mechanisms of robustness, exhibiting different patterns of language model adaptation, representational alignment, and prediction repair. Our results provide a more complete understanding of how synthetic surface variation improves robustness and offer practical guidance for selecting CPT strategies in multilingual and dialectal settings.

arXiv:2608.04586v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2TT data per language for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all $45 times 44$ translation directions. Our 4B-parameter model achieves state-of-the-art performance, outperforming substantially larger baselines. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby breaking the curse of multilinguality without compromising high-resource performance. To support future multilingual S2TT research, we release our code and models.

Long-horizon agents accumulate context faster than they resolve tasks. Every tool output, observation, and intermediate reasoning step stays in the window, and the two capabilities that matter — holding that context and staying coherent across it — have so far been available almost exclusively from cloud endpoints. That excludes regulated industries, public-sector institutions, and on-device applications, where the data is not permitted to leave the boundary at all. Pokee AI released Pokee-Isaac 28B, a 28B text-only foundation model with a 10M-token context window, designed to run inside that boundary. The Pokee research team claims 93.3% on RULER at 10M tokens, parity with the strongest cost-optimized cloud baselines on agentic benchmarks, and a serving profile that fits a single GPU. Is it deployable Yes — but licensed, not open-weight. Pokee AI serves Isaac through an OpenAI-compatible developer API, and licenses it for deployment inside a VPC, on-premises, or on-device. The launch announcement advertises Day-0 support for vLLM and SGLang, and single-GPU serving starting from an RTX 4090 or equivalent. The research team publishes measurements only from a single B200-class GPU, so treat the consumer-GPU claim as vendor guidance rather than a reported result. Company level: This fits organizations that already own their inference stack — mid-size and enterprise teams with a platform group, plus device OEMs. A solo practitioner without on-prem hardware should use the hosted API instead; the boundary argument only pays off if you have a boundary. Industries: Healthcare and payors, financial services and insurance, defense and public sector, legal and e-discovery, and pharma or semiconductor R&D. The common trait is a rule that the data cannot cross an external API boundary, not a preference for privacy. Applications: Whole-repository code review, multi-year contract and claims analysis, incident forensics over full log archives, and long-running tool agents that never need summarization or context pruning. The research paper makes this second point explicitly: when enough usable context is available in-boundary, memory hierarchies and compression become optional rather than required. Long-context results On RULER, Isaac stays above 93.3% at every tested length, ending at 93.3% at 10M. GPT-5.6 Luna and Gemini 3.5 Flash Lite track it to 512K, then hit context-overflow at 1M. On MRCR v2 with 8 needles, Isaac scores 0.607, 0.743, and 0.500 at 256K, 512K, and 1M. Its margin over Gemini widens from 0.133 to 0.295 across that sweep. Agentic and security results Isaac leads BFCL v4 at 70.94 against Luna’s 70.61. The report calls that parity rather than a lead, which is the correct read. On τ³-bench it averages 0.662 across four domains, ahead of Gemini’s 0.631, with banking at 0.186 for everyone’s difficulty. On MCP-Atlas it places third at 74.59% coverage, but uses 9.10 turns per task against Gemini’s 14.99. On Terminal-Bench 2.1 it resolves 56 of 86 text-compatible tasks (65.1%), behind Luna’s 60. That is the one benchmark a cloud baseline wins, and the report states it plainly. On DTAP red-teaming, Isaac records the lowest direct (36.0), indirect (35.2), and combined (35.6) attack success rates, with 82.5 benign success. One condition differs: baselines ran under the stock runner, Isaac under the Pokee harness. Efficiency, pricing, and portability Under the RULER workload on one B200-class GPU, TTFT is 23.6s at 1M and 72.9s at 10M. Prefill throughput rises with context, from 42,400 to 137,200 tokens/s, so a ten-fold longer prompt costs roughly three times the TTFT. List pricing is $0.15/$1.00 per million input/output tokens, marked provisional. Isaac also runs fully on-device on Intel Arc Pro B70 and Core Ultra Series 3 (Panther Lake), and on Qualcomm Snapdragon X2 Elite. Key Takeaways Pokee-Isaac 28B scores 93.3% on RULER at 10M tokens; every baseline in its panel returns 0.0 beyond 2M. Prefill reaches 137,200 tokens/s at 10M context on one B200; decode holds flat near 335 tokens/s. It leads BFCL v4 (70.94) and τ³-bench (0.662 avg), places second on Terminal-Bench 2.1, third on MCP-Atlas. Lowest combined attack success rate on DTAP (35.6) while keeping 82.5 benign task success. Weights are not published; deployment is licensed into VPC, on-premises, or on-device. Check out the Blog and Paper. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Pokee AI Releases Pokee-Isaac 28B: A 10M-Token Context Agentic Model Built to Run Inside the Customer Boundary appeared first on MarkTechPost.

Microsoft has open sourced code-testing-generator, a polyglot agent that writes unit tests and then proves they work. It ships in the dotnet-test plugin inside the MIT-licensed dotnet/skills repository. The agent targets a gap that coding assistants usually leave open. A prompt like ‘generate unit tests’ does not say which framework, file location or assertions to use. code-testing-generator settles those decisions by reading the repository before it writes anything. It then plans, writes, runs and checks the tests it produces. On Microsoft’s internal 152-task benchmark, it completed 140 tasks against 120 for stock GitHub Copilot. Both setups used the same model and prompts. Is it deployable Yes. It is an agent definition along with skills, not a hosted service, so it runs inside your existing coding agent and code stays local. Company stage: viable from solo maintainers upward. Startups and mid-market teams gain most, because the agent supplies repository research a small team has no time to encode. Enterprises can fork the language guidance to match internal frameworks. Industries: regulated or audit-heavy software estates — financial services, healthcare, insurance, public sector — plus platform teams paying down legacy test debt. Applications: backfilling tests on untested modules, generating tests for a pull-request diff, raising coverage before a release gate, and standardising conventions across polyglot monorepos. What the agent actually does It coordinates work through a Research-Plan-Implement (RPI) pipeline. It searches the repository for code needing tests, detects the language and test framework, reads existing tests for conventions, and finds the real build and test commands. That last step targets a specific failure: a test project that builds locally but never runs in CI because nothing registered it. The agent then picks one of three strategies. Direct writes and validates tests immediately. Single pass runs one cycle. Iterative repeats it for large scopes or coverage targets. It never modifies production code, and avoids tests that call external URLs, bind ports or depend on timing. The verification gate Before reporting completion, the agent runs five checks. It reasons about small code changes that should make the tests fail, a lightweight form of mutation testing. It looks for weak or missing assertions. It maps every requested scenario to a test. It builds the full workspace and runs the full suite. It confirms the repository’s own test command discovers the new tests. Benchmark results On Microsoft’s internal benchmark of 152 tasks from real repositories, the agent completed 140 (92.1%) versus 120 (78.9%) for stock GitHub Copilot on the same model and prompts (63% fewer failures). The gain is concentrated. On 89 vague prompts, the agent resolved 79 (88.8%) against 59 (66.3%), cutting failures from 30 to 10. On 63 detailed prompts, both scored 61 (96.8%). On 15 tasks targeting a specific diff, the agent passed all 15 and stock Copilot passed none. Notably, the agent generated 2.3% fewer tests (6,963 vs 7,129) at effectively identical line coverage (72.4% vs 72.2%). Average task time was 359 seconds against 380. Token use per completed task was 3.2% higher. On 45 .NET tasks, Claude Opus 4.8 reached 43/45 with the agent versus 35/45 stock; GPT-5.5 reached 41/45 versus 36/45. On the harder external SWE Atlas benchmark, completion was 16/44 versus 12/44. Explainer: how the agent turns one prompt into verified tests Key Takeaways Open source, MIT-licensed, polyglot unit-test agent from Microsoft’s .NET team. Research-Plan-Implement pipeline replaces one-shot generation with repository-aware planning. 92.1% vs 78.9% task completion against stock Copilot on the same model. Gains come almost entirely from vague prompts and diff-targeted requests. Fewer tests, same coverage, 5.5% faster — reliability, not volume. Check out the Technical details and Repo. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Microsoft Open Sources code-testing-generator: a Polyglot Unit-Test Agent That Hits 92.1% Task Completion Versus 78.9% for Stock Copilot appeared first on MarkTechPost.

arXiv:2608.05576v1 Announce Type: new Abstract: When a large language model (LLM) writes Harry Potter fanfiction, it reliably produces fundamental elements of the Hogwarts universe, such as recognizable places and characters. Human-written Harry Potter fanfictions, however, typically include these fundamentals and much more, incorporating stylistically irregular content and relationship-diverse plotlines. This gap between LLM and human writing has been noted across a variety of domains. LLMs tend to produce “average” writing, while human writing contains more diverse content that covers a broader distribution. Existing work has shown the existence of this distributional “gap”, but no work has proposed a systematic way to measure it. Our paper proposes a human-grounded framework that uses the empirical distribution of human writing on a topic to measure the distributional breadth of LLM-generated content on that same topic. We propose two metrics, LLM Coverage (LLM-Cov) and In-Boundary Rate (IBR), that separate the plausibility of LLM content from its distributional breadth. Across ideation and narrative tasks, we find that current LLMs produce plausible but narrow content that concentrates near the center of the human response space. Our framework can enable researchers to better assess the distributional breadth of LLM-authored content, which we term its “cultural reach”.