Committee

Meta has released Muse Glimmer, a 30-billion-parameter multimodal model distilled from Muse Spark. It is tuned for always-on local agent workflows, and ships under Apache 2.0. A 30B model normally needs over 55 GB of memory at full precision. Meta compresses it to roughly 4-bit, then adds block-level speculative decoding so it answers fast enough to sit inside a real agent loop. The result runs on one consumer GPU or a Mac, with no network call. Is it deployable? Yes, the weights are open under Apache 2.0. The Hugging Face collection carries BF16 weights, GGUF k-quants, ExecuTorch builds, and the DFlash drafter. Self-hosting is the day-one path. Which companies: Solo developers and startups can run it on one 24 GB GPU or an M4/M5 Max Mac. Mid-market teams get on-prem inference without a per-token bill. Regulated enterprises get an air-gappable agent. Meta advises adding system-level guardrails rather than shipping the model as a bare endpoint. Industries: Healthcare, legal, financial services, defense and public sector, manufacturing, and field service. These are the settings where data residency, offline operation, or latency rule out a cloud call. Applications: Desktop agents that read screenshots, coding agents, and schema-based function calling. Also document and chart understanding, synthetic data generation, and LLM-as-a-judge evaluation. Model and training Muse Glimmer is a dense causal transformer with a dedicated perception encoder. Total parameters are roughly 30B, including the vision tower. Grouped-query attention uses 32 query heads and 2 KV heads. Attention repeats a [Local, Local, Local, Global] pattern with a 2,048 sliding window. RoPE is applied to local layers only, with theta 500,000. The vision side is a ~1.8B ViT-G/14 perception encoder accepting up to 4,096 visual tokens per image. Context length is 131,072+, vocabulary is 202,048 tokens, and the knowledge cutoff is January 4, 2026. Input is text and image; output is text. Audio is not supported, and video is processed as individual frames. Training ran in three phases: Pre-training used logit distillation on Muse Spark’s outputs. Mid-training added longer-context, agent-heavy data with richer reasoning traces. Post-training combined supervised fine-tuning with on-policy distillation and reinforcement learning across general, reasoning, coding, and agentic domains. Fitting 30B onto consumer hardware At full precision the model needs over 55 GB of memory. Meta compresses weights to approximately 4-bit precision, which brings the language model under 20 GB. That leaves headroom inside a 24 GB or 32 GB envelope. The KV cache, perception encoder, and drafter share it. Two quantized builds ship. K-Quant-Dynamic targets 32 GB VRAM at 0.2% average degradation. K-Quant-17GB targets 24 GB VRAM at 1.0%. Degradation is averaged over accuracy metrics across 15 common benchmarks. Generation speed comes from DFlash, a block-diffusion drafter that predicts 16 tokens in one forward pass. The main model verifies the block in parallel. The drafter uses 5 layers, sliding-window attention at 2,048, and 32 query / 8 KV heads. Meta measured K-Quant-17GB at batch size 1 with greedy decoding. On an RTX 5090, throughput rises from 74.9 to 233.4 tok/s, a 3.1x speedup. Apple M5 Max moves from 26.6 to 50.2 tok/s, and M4 Max from 23.7 to 37.8 tok/s. Benchmarks Meta compares Muse Glimmer against Gemma4-31B and Qwen3.6-27B in thinking mode. It leads on MCP Atlas at 75.5, against 54.2 and 62.5. It also leads on DeepSearch QA at 74.6, Gaia2 at 43.3, and SWE-Bench Pro at 51.2. Reasoning scores follow: AIME 2026 at 94.7, IFBench at 77.0, AA-LCR at 80.0. Qwen3.6-27B stays ahead on OSWorld-Verified, 75.6 versus 65.9. It also leads TerminalBench 2.1 at 60.7 and SWE-Bench Verified at 77.2. The pattern is consistent. Muse Glimmer wins on agentic orchestration and reasoning. It trails on computer-use and terminal work. On safety, Siren AgentDojo attack success rate is 28.4 with utility 94.2. Meta states the model does not meet the Frontier AI definition in its Advanced AI Scaling Framework. It rates chem/bio, cyber, and loss-of-control risk at moderate or lower. Key Takeaways 30B open-weights agentic model, Apache 2.0, distilled from Muse Spark. 4-bit quantization fits it in 24 GB VRAM at 1.0% degradation. DFlash 16-token block speculation gives 3.1x decode speedup on RTX 5090. Beats both comparators on MCP Atlas, DeepSearch QA, and SWE-Bench Pro. Trails Qwen3.6-27B on OSWorld-Verified and TerminalBench 2.1. Check out the Model weights on HF, Details and Meta Blog. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Meta AI Releases Muse Glimmer: A 30B Open-Weights Agentic Model That Runs on One Consumer GPU appeared first on MarkTechPost.

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here. Last week, I headed 30 miles south of San Francisco to a hotel in Mountain View, California, to join some of the most accomplished, and some of the most promising, AI researchers in the world. I was hosting roundtable interviews and speaking at a media training for a convening of the Schmidt Sciences AI2050 program, an initiative funded by Eric and Wendy Schmidt that supports academics whose work involves AI. The fellows list is a who’s who of AI luminaries, and though not all of them made it out to the Bay, every time I turned a corner I saw a scientist whom I’d interviewed previously or whose research I admired. (Full disclosure: I received a science communication award funded by Schmidt Sciences in 2024.)  It’s a weird time for university AI researchers, who make up most of the AI2050 group. In the past four years, AI research has reoriented around large language models, and its cutting edge has moved from academic institutions to private companies. Universities simply can’t afford the GPUs required to train and run frontier models, and even if they could, Anthropic and OpenAI aren’t letting anyone else see the inner details of Claude or ChatGPT. In a conversation over lunch, Nika Haghtalab, a computer science professor at UC Berkeley, said that being an AI academic these days was like being a biologist in a world in which private companies had exclusive control over the gene-editing tool CRISPR. Experts outside the frontier labs can study how ChatGPT and Claude behave, but they can’t do any detailed research on the design and training of those tools, nor can they steer that design or training themselves.  The AI2050 program does offer fellows some funding that they can use to buy GPUs, which some researchers I spoke with said was a major benefit of participating in the program. But money remains a pressing concern, especially given the reduction of federal scientific funding in the United States. Even for researchers who don’t run local models themselves, the cost of repeatedly querying OpenAI’s, Anthropic’s, and Google’s models in order to study them rigorously can be prohibitive. Rather than focusing on advancing capabilities, many fellows aim their attention at questions that are unlikely to be addressed by Anthropic or OpenAI. “I try not to work on problems that I think are gonna be solved by a tech company,” says Anjalie Field, a computer science professor at Johns Hopkins. Companies need to make money, and research questions that have little promise of profit might not be worth investing in—especially if their answers might make the companies look bad. Recently, for example, Field conducted a study in which she found that language models give less sophisticated responses to prompts that are phrased in ways more commonly used by women than by men. It’s difficult to imagine that kind of research coming out of Anthropic or OpenAI. There’s also a huge group of AI academics who don’t work with LLMs at all. Many of them are scientists who build specialized AI models that can analyze data, make useful predictions, or even simulate entire physical systems. Those researchers aren’t necessarily competing with the frontier labs—Google DeepMind’s AlphaFold team, which built a Nobel Prize–winning model that predicts the structures of proteins, was disbanded last month. But they face plenty of their own challenges. At the convening, several voiced concerns about how the widespread ignorance of non-LLM AI was affecting their work. Researchers who build specialized AI tools to help address climate change, for example, sometimes struggle to advocate for their work when so many people believe that “AI” means “energy-guzzling LLMs.” All these challenges are changing the landscape of academia: Several prominent academics have recently taken leave from their universities to join frontier labs, and many AI2050 fellows hold industry positions alongside their academic jobs. And in the past six months, yet another threat has emerged. OpenAI’s models have solved a number of real research problems in mathematics, and some experts are worried that humans might not have a future in pure math. One fellow I spoke with said that she was concerned about the mental health of her mathematician peers. But it’s not all doom and gloom. For one thing, empirical science may prove much more difficult to automate than mathematics, because collecting data is an intrinsically slow process. And some researchers see AI mathematicians and scientists as a boon rather than a threat—including Tim Dettmers, a computer scientist at Carnegie Mellon who works to make AI models faster and cheaper to run. AI scientists won’t replace humans, Dettmers says. On the contrary, they could make human scientists far more efficient, so that he and his peers have the chance to pursue all the wild and inspired ideas they might otherwise never have gotten around to. And scientists are a resilient sort. The very resource constraints that prevent them from training frontier models also push them to discover new ways to make models smaller and more efficient, or to explore completely new architectures. If the next big AI breakthrough comes not from a major company but from a scrappy academic lab, I won’t be shocked.

In this article, you will learn how prompt caching and fine-tuning differ as strategies for reducing cost and latency in agentic AI systems, and how…

Long-running agents accumulate state that no transcript captures. A coding agent at step 10 holds edited files, a running dev server, installed packages, and a warm prompt cache. When it misreads a traceback and rewrites a file that was already correct, neither available recovery path is cheap: patching forward grows the context and the token bill, and restarting from step one re-pays every model and tool call while reproducing nothing exactly, because runs are non-deterministic. Jumping back to step eight is the option engineers actually want, and it is the one existing runtimes cannot offer. Git versions files, not a live process or a cache. Researchers at Northeastern University and Stanford University have released Shepherd, a Python runtime substrate that records an agent run as a Git-like trace of typed events, so any past state can be forked and replayed. The research team reports forks 5× faster than Docker and over 95% prompt-cache reuse on replay. Is it deployable? Yes but it is available in early alpha and not ready for production. Shepherd is MIT-licensed and installable with pip install shepherd-ai from PyPI. It needs Python 3.11+. OS-level grant enforcement runs on macOS (Seatbelt) and Linux (Landlock, in a privileged container). Industries: Software engineering and DevOps, AI infrastructure and agent-platform vendors, quantitative finance research, security tooling and offensive-security research, and data engineering. The common trait is not the vertical. It is long-horizon agent runs against heavy sandbox state, where a failed run is expensive to redo. Applications: Live supervision of coding agents, with a meta-agent reverting a bad write before it commits. Automated recovery from a wrong tool call, without a full restart. Branching exploration over candidate agent strategies, compared side by side. Rollout generation for reinforcement learning, forking at selected turns. What Shepherd changes Shepherd is a Python substrate that records an agent’s execution as a first-class object. Every agent-environment interaction becomes a typed event in a Git-like execution trace. Core operations are formalized as functions and mechanized in Lean. Each interaction is effectively a commit. Unlike Git, the commit covers the agent process and the filesystem together, copy-on-write. A branch therefore carries live state, not just files. Returning to an earlier point is a single fork from that commit. The research team reports that Shepherd forks the agent process and its filesystem 5× faster than Docker. Because the prompt prefix through the branch point is unchanged, replay achieves over 95% prompt-cache reuse. The documentation organizes the framework around four concepts: tasks, effects, runs, and workspaces. A task is a typed function whose body the model fills in, so the signature is the contract. An effect is every crossing of the task boundary, and it can be watched, answered, or refused. A run is the durable record of those crossings. Permissions are declared in the signature. A May[GitRepo, ReadOnly] binding is compiled to that run’s writable roots and enforced at the native syscall jail. Once a run is forkable, a meta-agent can sit on top Forking enables higher-order agents that observe a trace and intervene before a bad write commits. The research team demonstrates three applications: In runtime intervention, a live supervisor raised pair-coding pass rates on CooperBench from 28.8% to 54.7%. In counterfactual meta-optimization, branching exploration beat baselines across four benchmarks by up to 11 points, while cutting wall-clock time by up to 58%. In Tree-RL training, forking rollouts at selected turns improved TerminalBench-2 from 34.2% to 39.4%. Key Takeaways Shepherd records each agent-environment interaction as a typed event in a Git-like, forkable trace. A commit covers the agent process and filesystem together, so a rewind restores live state, not just files. Reported: 5× faster forks than Docker and over 95% prompt-cache reuse on replay. A live supervisor lifted CooperBench pair-coding pass rates from 28.8% to 54.7%. Check out the Paper, GitHub, Experiments repo, Project page and PyPI. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Meet Shepherd: An Open-Source Python Substrate That Lets Meta-Agents Fork, Replay, and Revert Any Agent Run appeared first on MarkTechPost.

In this tutorial, we develop an end-to-end sentiment analysis workflow using the Stanford NLP IMDb Large Movie Review Dataset and compare classical machine learning with parameter-efficient transformer fine-tuning. We begin by establishing a reproducible environment and auditing the dataset for class ordering, review-length skew, duplicate leakage, and preprocessing artifacts before training a strong TF-IDF and Logistic Regression baseline. We then fine-tune DistilBERT with LoRA through PEFT, evaluate it using accuracy, macro-F1, ROC-AUC, confusion matrices, and ROC curves, and examine threshold selection and probability calibration through Expected Calibration Error and reliability analysis. Beyond headline metrics, we investigate confident errors, performance across review lengths, word-level occlusion saliency, and head-versus-tail truncation to understand how the model reaches its predictions and where long-context limitations affect performance. Finally, we use the unlabeled IMDb split for confidence-based pseudo-labeling, compare the resulting semi-supervised model against our baseline, and save the merged transformer for reusable sentiment inference. Copy CodeCopiedUse a different Browser import importlib.util, subprocess, sys, os, time, random, warnings, inspect, hashlib warnings.filterwarnings(“ignore”) os.environ[“TOKENIZERS_PARALLELISM”] = “false” os.environ[“WANDB_DISABLED”] = “true” _REQUIRED = { “transformers”: “transformers”, “datasets”: “datasets”, “peft”: “peft”, “accelerate”: “accelerate”, “sklearn”: “scikit-learn”, } _missing = [pkg for mod, pkg in _REQUIRED.items() if importlib.util.find_spec(mod) is None] if _missing: print(f”Installing: {‘, ‘.join(_missing)} …”) subprocess.run([sys.executable, “-m”, “pip”, “install”, “-q”, *_missing], check=True) print(“Done. (If imports fail below, restart the runtime and re-run.)n”) import numpy as np import pandas as pd import torch import matplotlib.pyplot as plt from datasets import load_dataset from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline from sklearn.metrics import (accuracy_score, f1_score, roc_auc_score, classification_report, confusion_matrix, roc_curve) from transformers import (AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer, DataCollatorWithPadding, EarlyStoppingCallback, set_seed) from peft import LoraConfig, get_peft_model, TaskType def _disable_torchao_probe(): patched = [] try: import peft.import_utils as _piu _piu.is_torchao_available = lambda: False patched.append(“peft.import_utils”) except Exception: pass for _name, _mod in list(sys.modules.items()): if _name.startswith(“peft”) and hasattr(_mod, “is_torchao_available”): _mod.is_torchao_available = lambda: False patched.append(_name) return patched try: import torchao as _tao _v = getattr(_tao, “__version__”, “?”) if tuple(int(x) for x in _v.split(“.”)[:2]) < (0, 16): print(f”[compat] torchao {_v} < 0.16 -> disabling PEFT’s torchao probe: ” f”{‘, ‘.join(_disable_torchao_probe())}”) except Exception: _disable_torchao_probe() SEED = 42 MODEL_NAME = “distilbert-base-uncased” MAX_LEN = 256 N_TRAIN = 5000 N_EVAL = 2000 N_UNSUP = 3000 EPOCHS = 2 BATCH = 16 LR = 3e-4 FULL_RUN = False if FULL_RUN: N_TRAIN, N_EVAL, EPOCHS = 25000, 25000, 3 set_seed(SEED); random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED) DEVICE = “cuda” if torch.cuda.is_available() else “cpu” print(“=” * 79) print(f”device={DEVICE} | torch={torch.__version__} | ” f”gpu={torch.cuda.get_device_name(0) if DEVICE==’cuda’ else ‘n/a’}”) print(“=” * 79) t0 = time.time() raw = load_dataset(“stanfordnlp/imdb”) print(raw, f”nloaded in {time.time()-t0:.1f}sn”) print(“— example (truncated) —“) print(“label:”, raw[“train”][0][“label”], “|”, raw[“train”][0][“text”][:300], “…n”) first_labels = np.array(raw[“train”][“label”][:5]) last_labels = np.array(raw[“train”][“label”][-5:]) print(f”TRAP #1 – split ordering: first 5 labels {first_labels}, ” f”last 5 labels {last_labels} -> ALWAYS shuffle before subsampling.”) train_full = raw[“train”].shuffle(seed=SEED) test_full = raw[“test”].shuffle(seed=SEED) train_ds = train_full.select(range(min(N_TRAIN, len(train_full)))) eval_ds = test_full.select(range(min(N_EVAL, len(test_full)))) print(f” after shuffle+subsample: train balance = ” f”{np.bincount(train_ds[‘label’])}, eval balance = {np.bincount(eval_ds[‘label’])}”) lens = np.array([len(t.split()) for t in train_full[“text”]]) q = np.percentile(lens, [50, 75, 90, 95, 99]) print(f”nTRAP #2 – length (words): median={q[0]:.0f} p75={q[1]:.0f} p90={q[2]:.0f} ” f”p95={q[3]:.0f} p99={q[4]:.0f} max={lens.max()}”) print(f” ~{(lens > MAX_LEN*0.75).mean()*100:.1f}% of reviews exceed MAX_LEN={MAX_LEN} ” f”tokens (rough words->tokens factor 1.3). Section 9 measures what that costs.”) h_tr = {hashlib.md5(t.encode()).hexdigest() for t in raw[“train”][“text”]} h_te = {hashlib.md5(t.encode()).hexdigest() for t in raw[“test”][“text”]} print(f”nTRAP #3 – leakage: {len(h_tr & h_te)} exact duplicate reviews across ” f”train/test; {len(raw[‘train’])-len(h_tr)} dupes inside train itself.”) def clean(t): return t.replace(“<br />”, ” “).replace(“<br/>”, ” “).strip() plt.figure(figsize=(11, 3.2)) plt.subplot(1, 2, 1) plt.hist(np.clip(lens, 0, 1000), bins=60) plt.axvline(MAX_LEN, ls=”–“, color=”k”, label=f”MAX_LEN={MAX_LEN}”) plt.title(“Review length (words, clipped at 1000)”); plt.legend() plt.subplot(1, 2, 2) plt.bar([“neg”, “pos”], np.bincount(raw[“train”][“label”])) plt.title(“Train class balance (perfectly balanced)”) plt.tight_layout(); plt.show() We configure the Colab environment, install the required libraries, apply the PEFT–torchao compatibility fix, and set deterministic seeds for reproducible experiments. We load the Stanford IMDb dataset, shuffle and subsample the train and test splits, and inspect class balance, review-length distributions, duplicate leakage, and HTML artifacts. We also visualize review lengths and label frequencies so we understand the dataset structure before building any models. Copy CodeCopiedUse a different Browser print(“n” + “=” * 79 + “n3. TF-IDF BASELINEn” + “=” * 79) Xtr = [clean(t) for t in train_ds[“text”]]; ytr = np.array(train_ds[“label”]) Xte = [clean(t) for t in eval_ds[“text”]]; yte = np.array(eval_ds[“label”]) t0 = time.time() tfidf_clf = make_pipeline( TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_features=300_000, sublinear_tf=True, strip_accents=”unicode”), LogisticRegression(C=8.0, max_iter=2000, n_jobs=-1), ) tfidf_clf.fit(Xtr, ytr) p_tfidf = tfidf_clf.predict_proba(Xte)[:, 1] acc_tfidf = accuracy_score(yte, p_tfidf > 0.5) auc_tfidf = roc_auc_score(yte, p_tfidf) print(f”trained in {time.time()-t0:.1f}s -> acc={acc_tfidf:.4f} auc={auc_tfidf:.4f}”) vec, lr = tfidf_clf.steps[0][1], tfidf_clf.steps[1][1] feats, coefs = np.array(vec.get_feature_names_out()), lr.coef_[0] order = np.argsort(coefs) print(“nmost NEGATIVE n-grams:”, “, “.join(feats[order[:12]])) print(“most POSITIVE n-grams:”, “, “.join(feats[order[-12:]][::-1])) print(“n” + “=” * 79 + “n4. LoRA FINE-TUNINGn” + “=” * 79) tok = AutoTokenizer.from_pretrained(MODEL_NAME) def tokenize(batch): return tok([clean(t) for t in batch[“text”]], truncation=True, max_length=MAX_LEN) tr_tok = (train_ds.map(tokenize, batched=True, remove_columns=[“text”]) .rename_column(“label”, “labels”)) ev_tok = (eval_ds.map(tokenize, batched=True, remove_columns=[“text”]) .rename_column(“label”, “labels”)) base = AutoModelForSequenceClassification.from_pretrained( MODEL_NAME, num_labels=2, id2label={0: “NEGATIVE”, 1: “POSITIVE”}, label2id={“NEGATIVE”: 0, “POSITIVE”: 1}, ) lora_cfg = LoraConfig( task_type=TaskType.SEQ_CLS, r=16, lora_alpha=32, lora_dropout=0.05, target_modules=[“q_lin”, “v_lin”], modules_to_save=[“pre_classifier”, “classifier”], ) try: model = get_peft_model(base, lora_cfg) except ImportError as e: _disable_torchao_probe() print(f”[compat] retrying after backend probe failure: {e}”) model = get_peft_model(base, lora_cfg) model.print_trainable_parameters() def compute_metrics(eval_pred): logits, labels = eval_pred probs = torch.softmax(torch.tensor(logits), dim=-1).numpy()[:, 1] preds = (probs > 0.5).astype(int) return {“accuracy”: accuracy_score(labels, preds), “f1_macro”: f1_score(labels, preds, average=”macro”), “roc_auc”: roc_auc_score(labels, probs)} _ta = inspect.signature(TrainingArguments.__init__).parameters _eval_key = “eval_strategy” if “eval_strategy” in _ta else “evaluation_strategy” ta_kwargs = dict( output_dir=”./imdb_lora”, learning_rate=LR, per_device_train_batch_size=BATCH, per_device_eval_batch_size=BATCH * 2, num_train_epochs=EPOCHS, weight_decay=0.01, warmup_ratio=0.06, logging_steps=50, save_strategy=”epoch”, save_total_limit=1, load_best_model_at_end=True, metric_for_best_model=”accuracy”, fp16=(DEVICE == “cuda”), report_to=”none”, seed=SEED, ) ta_kwargs[_eval_key] = “epoch” _tr = inspect.signature(Trainer.__init__).parameters _tok_key = “processing_class” if “processing_class” in _tr else “tokenizer” trainer = Trainer( model=model, args=TrainingArguments(**ta_kwargs), train_dataset=tr_tok, eval_dataset=ev_tok, data_collator=DataCollatorWithPadding(tok), compute_metrics=compute_metrics, callbacks=[EarlyStoppingCallback(early_stopping_patience=2)], **{_tok_key: tok}, ) t0 = time.time() trainer.train() print(f”nfine-tuned in {(time.time()-t0)/60:.1f} min”) We train a strong TF-IDF and Logistic Regression baseline and inspect the most influential positive and negative n-grams to establish an interpretable reference point. We then tokenize the IMDb reviews and configure DistilBERT with LoRA adapters that update only a small subset of model parameters while keeping the

LLM applications fail in ways traditional software does not. The same prompt can produce different outputs. A retrieval step can return the wrong document while every HTTP status reads 200. An agent can loop through fourteen tool calls, burn thousands of tokens, and deliver a confidently wrong answer. Standard application performance monitoring (APM) alone does not capture this semantic behavior — prompt and output quality, retrieval relevance, or agent-level reasoning traces. This is the gap LLM observability and evaluation platforms fill. They record every span of an LLM pipeline — prompts, completions, retrievals, tool calls, token counts, latencies, and costs — and then score outputs for quality using automated evaluators. In 2026, this category has moved from optional tooling to core infrastructure for any team running AI in production. The market data reflects the shift. The Business Research Company sizes the LLM observability platform market at $2.69 billion in 2026, up from $1.97 billion in 2025, and projects $9.26 billion by 2030 at a 36.2% forecast CAGR. Gartner predicts that by 2028, LLM observability investments will account for 50% of GenAI deployments, up from 15% in early 2026. LangChain’s State of Agent Engineering survey of 1,300+ professionals found that 57% of respondents now run agents in production. Nearly 89% have implemented observability for their agents. Evaluation lags behind: 52.4% run offline evaluations, 37.3% run online evaluations, and 29.5% report no evaluation at all. Quality was cited by 32% as the top barrier to production deployment. This article compares the leading platforms across three axes: tracing depth, evaluation capability, and production monitoring. Figures were checked against primary sources (company documentation, press pages, and announcements) as of August 2026; where only secondary reporting exists, it is linked and identified as such. Rankings and “best for” judgments are editorial assessments, not measured benchmarks. How the Category is Structured in 2026 The market has split into four camps, and understanding the split matters more than any individual feature list. AI-native observability platforms: Langfuse, LangSmith, Braintrust, Arize, Opik — treat the LLM trace as the primary object. They capture nested spans across agents, retrievers, and tools, and attach evaluation scores to production traffic. Open-source and source-available evaluation libraries and platforms: Arize Phoenix, DeepEval (Confident AI), MLflow, RAGAS — focus on scoring outputs: faithfulness, hallucination, answer relevance, and task completion, often via LLM-as-a-judge. AI gateways: Helicone, Portkey, LiteLLM — sit as a proxy between the application and model providers. They add logging, caching, cost tracking, and routing with minimal code changes. APM extensions: Datadog LLM Observability, New Relic, Dynatrace — bolt LLM tracing onto existing infrastructure monitoring so AI signals correlate with CPU, memory, and network metrics. One standard now connects all four camps. The OpenTelemetry GenAI semantic conventions define vendor-neutral gen_ai.* span attributes for model calls, token usage, agent steps, and tool executions. OpenTelemetry, a CNCF project, maintains these conventions, which are adopted by platforms including Google Cloud, AWS, Azure, and Datadog. The conventions now live in a dedicated repository, with the GenAI registry under active development as of August 2026. Coding agents are converging on the standard too: GitHub Copilot’s agent telemetry exposes gen_ai.* span trees, Claude Code provides opt-in OpenTelemetry tracing, and Codex includes native OpenTelemetry export support. Instrumenting once against gen_ai.* improves backend portability and reduces vendor-specific instrumentation, even if implementations still differ. Buyers in 2026 should treat OTel compatibility as a hard requirement, not a nice-to-have. The Three Axes: Tracing, Evals, and Production Monitoring Because vendors use these terms loosely, precise definitions help before comparing platforms: Tracing is the record of everything an LLM application did. A trace contains nested spans: the user input, each retrieval call, each model invocation with its exact prompt and parameters, each tool execution, and the final output. Depth matters because agent traces are deeply nested with heavy payloads — a single conversation can generate megabytes of data across dozens of runs and tool calls. Non-determinism makes tracing non-negotiable: the same prompt produces different outputs, so an issue cannot be reproduced without capturing the exact input, model parameters, and temperature at call time. Evals answer the question tracing cannot: was the output any good? Offline evals score curated datasets before deployment, catching regressions when a prompt, model, or retrieval index changes. Online evals score live production traffic, typically via LLM-as-a-judge, sampling traces and grading them for faithfulness, relevance, toxicity, or task completion. The hardest failures are outputs that are technically valid but wrong for the domain — a hallucinated policy, a drifting tone, a retrieval miss that produces a confident but incorrect answer. Traditional latency, error-rate, and availability metrics do not detect these semantic quality failures. Production monitoring closes the loop: dashboards, cost attribution per model and user, latency percentiles, drift detection across prompts and use cases, and alerting when quality scores fall. The best platforms feed production traces back into eval datasets, so every real-world failure becomes a future regression test. A platform can be strong on one axis and weak on another. Gateways excel at monitoring but skip deep tracing. Eval libraries score outputs but do not watch production. The platforms below are ranked on how completely they cover all three. 1. Langfuse (ClickHouse) Langfuse describes itself as the most widely adopted LLM engineering platform, and its open-source adoption numbers back a strong claim. Tracing: Langfuse captures nested traces for LLM calls, retrieval, embedding, and agent actions through OpenTelemetry, LangChain, OpenAI SDK, and LiteLLM integrations. Its signature nested trace view collapses a multi-step RAG or agent run into a stepable tree with per-span latencies and token counts. An observations-centric data model shipped in March 2026, delivering 10x+ dashboard performance gains and laying the groundwork for Langfuse v4, which the company says runs up to 165x faster. Evals: The platform supports LLM-as-a-judge evaluators, human annotation queues, custom scores, and dataset-based regression testing that runs in CI via GitHub Actions. Evaluator templates cover hallucination, toxicity, and relevance. Production monitoring: Cost breakdowns by model, user, or session, plus session replays for conversational agents. Deployment: MIT-licensed core,

Mistral AI has released Shieldstral 1.0 3B, an open-weights, policy-adaptive multimodal safety classifier that treats content moderation as a single yes/no question rather than a fixed taxonomy of harm categories. Most guardrail models bake their category list into the weights, so re-targeting one to a new deployment context means retraining — and the same content can be acceptable on a cybersecurity research tool while being harmful on a mental-health platform. Shieldstral inverts that: operators write the policy as a plain-language question at inference time, and the model returns a calibrated safety score from a single forward pass. Built on Ministral-3-3B-Base-2512 with a native Pixtral vision encoder and released under Apache 2.0, it reports 84.9% average F1 on text safety — matching GPT-OSS-Safeguard-20B — and 83.8% on multimodal safety, ahead of every baseline Mistral evaluated. Is it deployable? Yes, and locally. Shieldstral-1.0-3B fits in 16GB of VRAM in BF16, runs on a single GPU, and is licensed Apache 2.0 for commercial and non-commercial use. Serving paths are already in place: vLLM (≥0.26.0, recommended), llama.cpp via GGUF conversion with Q8_0/Q5_K_M/Q4_K_M quantization, SGLang, and Transformers — with fine-tuning supported through Axolotl. The classifier emits one token, so latency and cost sit far below reasoning-based guards like GPT-OSS-Safeguard-20B. Which level of company: the 16GB footprint puts it within reach of seed-stage AI product teams that cannot justify a moderation vendor contract, while the open license and self-hosting story suit mid-market and enterprise teams that need guardrails inside a VPC or on-prem for data-residency and audit reasons. Multi-tenant SaaS vendors get a specific win — one checkpoint can enforce a different policy per customer. Industries: consumer social and UGC platforms, ed-tech and child-safety surfaces, healthcare and mental-health apps, fintech and insurance support automation, gaming and voice chat, marketplaces and ad/creative review, and public sector deployments with sovereignty requirements. Applications: user-prompt moderation, model-response moderation, refusal classification, image-plus-caption review for ads and memes, training-data and RAG-corpus curation, output gating in agentic pipelines, and per-tenant policy enforcement. Because the output is a continuous score rather than a label, teams can tune the threshold per surface or route borderline scores to human review instead of hard-blocking. Moderation as a binary question Shieldstral reduces moderation to one yes/no question. A fixed system message establishes the task; the user message carries three fields: <Instruct> (evaluation context and strictness), <Query> (the policy, phrased as a single yes/no question), and <Document> (a prompt, a response, a prompt–response pair, or an image with optional text). At inference the model unembeds only toward the yes and no token IDs and softmax-normalizes them into a continuous score, thresholded at τ=0.5. That collapses prompt classification, response moderation, refusal detection, and toxicity detection into one problem — and it means the policy lives entirely in the prompt. Mistral’s guidance is one policy per call; for a broad safe/unsafe verdict, list the categories in <Instruct> and ask a single wide <Query>. The data recipe The claimed advantage comes from data, not scale: roughly 54.1M samples — 45.2M open-source text, 4.4M synthetic contrastive text, 4.5M multimodal. A template-based unification layer converts every dataset into the same instruction–query–document format via per-dataset processors, with randomized phrasings and calibrated strictness (strict for adversarial jailbreaks, lenient for response-quality data). The more interesting piece is contrastive generation. An LLM rewrites safe text into an unsafe variant that violates a target category but deliberately not its sibling, producing a positive and a hard negative over identical content in one call. That teaches the model which policy is violated rather than a coarse safe/unsafe split. Image data — which cannot be synthesized the way text can — is supplemented with general-purpose image datasets as negatives, query mutation across a 14-subcategory visual taxonomy, and vision–language reranker filtering. Training is LoRA fine-tuning followed by a three-way SLERP merge: 0.6 public+generated, 0.3 public-only, 0.1 Ministral-3B-Instruct. Results On text safety, Shieldstral reports 84.9% average F1, tying GPT-OSS-Safeguard-20B (84.9%) as the smallest model in the comparison, with wins on ToxicChat (84.1), HarmBench (99.4), and Aegis v2 response (87.2). On multimodal safety it reports 83.8% overall versus 77.6% for OmniGuard-7B, leading VLGuard (97.7) and UnsafeBench (81.8); LlavaGuard-7B still leads its namesake benchmark at 81.4. On the adaptability benchmark — built on a deliberately divergent taxonomy of 12 super classes, 26 subcategories, and 52 leaf categories with 90 fixed queries, where no leaf maps one-to-one to training — Shieldstral scores 91.3% F1, behind GPT-OSS-Safeguard-20B (94.1%) and Nemotron-3.5-Safety-4B (91.8%), but without generating a reasoning trace. Refusal detection lands at 91.5% overall against 93.7% for GPT-OSS-Safeguard-20B. Where it is weaker: multilingual prompt classification lags on Arabic and Indonesian and on RTP-LX prompts (70.3 vs 86.1 for Nemotron-3.5-Safety-4B). Mistral also flags reduced reliability on adversarial or obfuscated inputs and very long documents. Trained context is 32k tokens across 12 languages. Key Takeaways 3B Apache 2.0 multimodal guardrail; policy is a plain-language question at inference time, no retraining. 84.9% text F1 ties a 20B model; 83.8% multimodal F1 is best-in-class among evaluated baselines. 54.1M samples with sibling-contrastive rewrites is the actual mechanism behind policy generalization. Single forward pass, single token out, continuous score at τ=0.5 — cheap enough for real-time gating. Weak spots: low-resource languages, obfuscated inputs, long documents. Check out the Paper, Model on Hugging Face, and Technical Details. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Mistral AI Releases Shieldstral 1.0 3B: An Open-Weights Policy-Adaptive Multimodal Safety Classifier Matching Models 7× Its Size appeared first on MarkTechPost.

arXiv:2608.05510v1 Announce Type: new Abstract: Dialectal variation remains a major challenge for multilingual language models. Perturbation-based continued pre-training (CPT) has emerged as a promising approach to improving robustness, yet existing work largely evaluates individual perturbation strategies in isolation and provides limited insight into why they work. We present a systematic study of perturbation-based CPT for multilingual dialect robustness in LLMs, comparing six training conditions across nine German, Italian, and Arabic dialect tasks. Perturbation-based CPT, especially character-noised CPT, consistently improves zero-shot dialect robustness while largely preserving standard variety performance. More importantly, we show that methods with similar downstream performance induce distinct mechanisms of robustness, exhibiting different patterns of language model adaptation, representational alignment, and prediction repair. Our results provide a more complete understanding of how synthetic surface variation improves robustness and offer practical guidance for selecting CPT strategies in multilingual and dialectal settings.

arXiv:2608.04586v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2TT data per language for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all $45 times 44$ translation directions. Our 4B-parameter model achieves state-of-the-art performance, outperforming substantially larger baselines. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby breaking the curse of multilinguality without compromising high-resource performance. To support future multilingual S2TT research, we release our code and models.

Long-horizon agents accumulate context faster than they resolve tasks. Every tool output, observation, and intermediate reasoning step stays in the window, and the two capabilities that matter — holding that context and staying coherent across it — have so far been available almost exclusively from cloud endpoints. That excludes regulated industries, public-sector institutions, and on-device applications, where the data is not permitted to leave the boundary at all. Pokee AI released Pokee-Isaac 28B, a 28B text-only foundation model with a 10M-token context window, designed to run inside that boundary. The Pokee research team claims 93.3% on RULER at 10M tokens, parity with the strongest cost-optimized cloud baselines on agentic benchmarks, and a serving profile that fits a single GPU. Is it deployable Yes — but licensed, not open-weight. Pokee AI serves Isaac through an OpenAI-compatible developer API, and licenses it for deployment inside a VPC, on-premises, or on-device. The launch announcement advertises Day-0 support for vLLM and SGLang, and single-GPU serving starting from an RTX 4090 or equivalent. The research team publishes measurements only from a single B200-class GPU, so treat the consumer-GPU claim as vendor guidance rather than a reported result. Company level: This fits organizations that already own their inference stack — mid-size and enterprise teams with a platform group, plus device OEMs. A solo practitioner without on-prem hardware should use the hosted API instead; the boundary argument only pays off if you have a boundary. Industries: Healthcare and payors, financial services and insurance, defense and public sector, legal and e-discovery, and pharma or semiconductor R&D. The common trait is a rule that the data cannot cross an external API boundary, not a preference for privacy. Applications: Whole-repository code review, multi-year contract and claims analysis, incident forensics over full log archives, and long-running tool agents that never need summarization or context pruning. The research paper makes this second point explicitly: when enough usable context is available in-boundary, memory hierarchies and compression become optional rather than required. Long-context results On RULER, Isaac stays above 93.3% at every tested length, ending at 93.3% at 10M. GPT-5.6 Luna and Gemini 3.5 Flash Lite track it to 512K, then hit context-overflow at 1M. On MRCR v2 with 8 needles, Isaac scores 0.607, 0.743, and 0.500 at 256K, 512K, and 1M. Its margin over Gemini widens from 0.133 to 0.295 across that sweep. Agentic and security results Isaac leads BFCL v4 at 70.94 against Luna’s 70.61. The report calls that parity rather than a lead, which is the correct read. On τ³-bench it averages 0.662 across four domains, ahead of Gemini’s 0.631, with banking at 0.186 for everyone’s difficulty. On MCP-Atlas it places third at 74.59% coverage, but uses 9.10 turns per task against Gemini’s 14.99. On Terminal-Bench 2.1 it resolves 56 of 86 text-compatible tasks (65.1%), behind Luna’s 60. That is the one benchmark a cloud baseline wins, and the report states it plainly. On DTAP red-teaming, Isaac records the lowest direct (36.0), indirect (35.2), and combined (35.6) attack success rates, with 82.5 benign success. One condition differs: baselines ran under the stock runner, Isaac under the Pokee harness. Efficiency, pricing, and portability Under the RULER workload on one B200-class GPU, TTFT is 23.6s at 1M and 72.9s at 10M. Prefill throughput rises with context, from 42,400 to 137,200 tokens/s, so a ten-fold longer prompt costs roughly three times the TTFT. List pricing is $0.15/$1.00 per million input/output tokens, marked provisional. Isaac also runs fully on-device on Intel Arc Pro B70 and Core Ultra Series 3 (Panther Lake), and on Qualcomm Snapdragon X2 Elite. Key Takeaways Pokee-Isaac 28B scores 93.3% on RULER at 10M tokens; every baseline in its panel returns 0.0 beyond 2M. Prefill reaches 137,200 tokens/s at 10M context on one B200; decode holds flat near 335 tokens/s. It leads BFCL v4 (70.94) and τ³-bench (0.662 avg), places second on Terminal-Bench 2.1, third on MCP-Atlas. Lowest combined attack success rate on DTAP (35.6) while keeping 82.5 benign task success. Weights are not published; deployment is licensed into VPC, on-premises, or on-device. Check out the Blog and Paper. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Pokee AI Releases Pokee-Isaac 28B: A 10M-Token Context Agentic Model Built to Run Inside the Customer Boundary appeared first on MarkTechPost.