Uncategorized

Mistral AI has released Shieldstral 1.0 3B, an open-weights, policy-adaptive multimodal safety classifier that treats content moderation as a single yes/no question rather than a fixed taxonomy of harm categories. Most guardrail models bake their category list into the weights, so re-targeting one to a new deployment context means retraining — and the same content can be acceptable on a cybersecurity research tool while being harmful on a mental-health platform. Shieldstral inverts that: operators write the policy as a plain-language question at inference time, and the model returns a calibrated safety score from a single forward pass. Built on Ministral-3-3B-Base-2512 with a native Pixtral vision encoder and released under Apache 2.0, it reports 84.9% average F1 on text safety — matching GPT-OSS-Safeguard-20B — and 83.8% on multimodal safety, ahead of every baseline Mistral evaluated. Is it deployable? Yes, and locally. Shieldstral-1.0-3B fits in 16GB of VRAM in BF16, runs on a single GPU, and is licensed Apache 2.0 for commercial and non-commercial use. Serving paths are already in place: vLLM (≥0.26.0, recommended), llama.cpp via GGUF conversion with Q8_0/Q5_K_M/Q4_K_M quantization, SGLang, and Transformers — with fine-tuning supported through Axolotl. The classifier emits one token, so latency and cost sit far below reasoning-based guards like GPT-OSS-Safeguard-20B. Which level of company: the 16GB footprint puts it within reach of seed-stage AI product teams that cannot justify a moderation vendor contract, while the open license and self-hosting story suit mid-market and enterprise teams that need guardrails inside a VPC or on-prem for data-residency and audit reasons. Multi-tenant SaaS vendors get a specific win — one checkpoint can enforce a different policy per customer. Industries: consumer social and UGC platforms, ed-tech and child-safety surfaces, healthcare and mental-health apps, fintech and insurance support automation, gaming and voice chat, marketplaces and ad/creative review, and public sector deployments with sovereignty requirements. Applications: user-prompt moderation, model-response moderation, refusal classification, image-plus-caption review for ads and memes, training-data and RAG-corpus curation, output gating in agentic pipelines, and per-tenant policy enforcement. Because the output is a continuous score rather than a label, teams can tune the threshold per surface or route borderline scores to human review instead of hard-blocking. Moderation as a binary question Shieldstral reduces moderation to one yes/no question. A fixed system message establishes the task; the user message carries three fields: <Instruct> (evaluation context and strictness), <Query> (the policy, phrased as a single yes/no question), and <Document> (a prompt, a response, a prompt–response pair, or an image with optional text). At inference the model unembeds only toward the yes and no token IDs and softmax-normalizes them into a continuous score, thresholded at τ=0.5. That collapses prompt classification, response moderation, refusal detection, and toxicity detection into one problem — and it means the policy lives entirely in the prompt. Mistral’s guidance is one policy per call; for a broad safe/unsafe verdict, list the categories in <Instruct> and ask a single wide <Query>. The data recipe The claimed advantage comes from data, not scale: roughly 54.1M samples — 45.2M open-source text, 4.4M synthetic contrastive text, 4.5M multimodal. A template-based unification layer converts every dataset into the same instruction–query–document format via per-dataset processors, with randomized phrasings and calibrated strictness (strict for adversarial jailbreaks, lenient for response-quality data). The more interesting piece is contrastive generation. An LLM rewrites safe text into an unsafe variant that violates a target category but deliberately not its sibling, producing a positive and a hard negative over identical content in one call. That teaches the model which policy is violated rather than a coarse safe/unsafe split. Image data — which cannot be synthesized the way text can — is supplemented with general-purpose image datasets as negatives, query mutation across a 14-subcategory visual taxonomy, and vision–language reranker filtering. Training is LoRA fine-tuning followed by a three-way SLERP merge: 0.6 public+generated, 0.3 public-only, 0.1 Ministral-3B-Instruct. Results On text safety, Shieldstral reports 84.9% average F1, tying GPT-OSS-Safeguard-20B (84.9%) as the smallest model in the comparison, with wins on ToxicChat (84.1), HarmBench (99.4), and Aegis v2 response (87.2). On multimodal safety it reports 83.8% overall versus 77.6% for OmniGuard-7B, leading VLGuard (97.7) and UnsafeBench (81.8); LlavaGuard-7B still leads its namesake benchmark at 81.4. On the adaptability benchmark — built on a deliberately divergent taxonomy of 12 super classes, 26 subcategories, and 52 leaf categories with 90 fixed queries, where no leaf maps one-to-one to training — Shieldstral scores 91.3% F1, behind GPT-OSS-Safeguard-20B (94.1%) and Nemotron-3.5-Safety-4B (91.8%), but without generating a reasoning trace. Refusal detection lands at 91.5% overall against 93.7% for GPT-OSS-Safeguard-20B. Where it is weaker: multilingual prompt classification lags on Arabic and Indonesian and on RTP-LX prompts (70.3 vs 86.1 for Nemotron-3.5-Safety-4B). Mistral also flags reduced reliability on adversarial or obfuscated inputs and very long documents. Trained context is 32k tokens across 12 languages. Key Takeaways 3B Apache 2.0 multimodal guardrail; policy is a plain-language question at inference time, no retraining. 84.9% text F1 ties a 20B model; 83.8% multimodal F1 is best-in-class among evaluated baselines. 54.1M samples with sibling-contrastive rewrites is the actual mechanism behind policy generalization. Single forward pass, single token out, continuous score at τ=0.5 — cheap enough for real-time gating. Weak spots: low-resource languages, obfuscated inputs, long documents. Check out the Paper, Model on Hugging Face, and Technical Details. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Mistral AI Releases Shieldstral 1.0 3B: An Open-Weights Policy-Adaptive Multimodal Safety Classifier Matching Models 7× Its Size appeared first on MarkTechPost.

arXiv:2608.05510v1 Announce Type: new Abstract: Dialectal variation remains a major challenge for multilingual language models. Perturbation-based continued pre-training (CPT) has emerged as a promising approach to improving robustness, yet existing work largely evaluates individual perturbation strategies in isolation and provides limited insight into why they work. We present a systematic study of perturbation-based CPT for multilingual dialect robustness in LLMs, comparing six training conditions across nine German, Italian, and Arabic dialect tasks. Perturbation-based CPT, especially character-noised CPT, consistently improves zero-shot dialect robustness while largely preserving standard variety performance. More importantly, we show that methods with similar downstream performance induce distinct mechanisms of robustness, exhibiting different patterns of language model adaptation, representational alignment, and prediction repair. Our results provide a more complete understanding of how synthetic surface variation improves robustness and offer practical guidance for selecting CPT strategies in multilingual and dialectal settings.

arXiv:2608.04586v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2TT data per language for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all $45 times 44$ translation directions. Our 4B-parameter model achieves state-of-the-art performance, outperforming substantially larger baselines. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby breaking the curse of multilinguality without compromising high-resource performance. To support future multilingual S2TT research, we release our code and models.

Long-horizon agents accumulate context faster than they resolve tasks. Every tool output, observation, and intermediate reasoning step stays in the window, and the two capabilities that matter — holding that context and staying coherent across it — have so far been available almost exclusively from cloud endpoints. That excludes regulated industries, public-sector institutions, and on-device applications, where the data is not permitted to leave the boundary at all. Pokee AI released Pokee-Isaac 28B, a 28B text-only foundation model with a 10M-token context window, designed to run inside that boundary. The Pokee research team claims 93.3% on RULER at 10M tokens, parity with the strongest cost-optimized cloud baselines on agentic benchmarks, and a serving profile that fits a single GPU. Is it deployable Yes — but licensed, not open-weight. Pokee AI serves Isaac through an OpenAI-compatible developer API, and licenses it for deployment inside a VPC, on-premises, or on-device. The launch announcement advertises Day-0 support for vLLM and SGLang, and single-GPU serving starting from an RTX 4090 or equivalent. The research team publishes measurements only from a single B200-class GPU, so treat the consumer-GPU claim as vendor guidance rather than a reported result. Company level: This fits organizations that already own their inference stack — mid-size and enterprise teams with a platform group, plus device OEMs. A solo practitioner without on-prem hardware should use the hosted API instead; the boundary argument only pays off if you have a boundary. Industries: Healthcare and payors, financial services and insurance, defense and public sector, legal and e-discovery, and pharma or semiconductor R&D. The common trait is a rule that the data cannot cross an external API boundary, not a preference for privacy. Applications: Whole-repository code review, multi-year contract and claims analysis, incident forensics over full log archives, and long-running tool agents that never need summarization or context pruning. The research paper makes this second point explicitly: when enough usable context is available in-boundary, memory hierarchies and compression become optional rather than required. Long-context results On RULER, Isaac stays above 93.3% at every tested length, ending at 93.3% at 10M. GPT-5.6 Luna and Gemini 3.5 Flash Lite track it to 512K, then hit context-overflow at 1M. On MRCR v2 with 8 needles, Isaac scores 0.607, 0.743, and 0.500 at 256K, 512K, and 1M. Its margin over Gemini widens from 0.133 to 0.295 across that sweep. Agentic and security results Isaac leads BFCL v4 at 70.94 against Luna’s 70.61. The report calls that parity rather than a lead, which is the correct read. On τ³-bench it averages 0.662 across four domains, ahead of Gemini’s 0.631, with banking at 0.186 for everyone’s difficulty. On MCP-Atlas it places third at 74.59% coverage, but uses 9.10 turns per task against Gemini’s 14.99. On Terminal-Bench 2.1 it resolves 56 of 86 text-compatible tasks (65.1%), behind Luna’s 60. That is the one benchmark a cloud baseline wins, and the report states it plainly. On DTAP red-teaming, Isaac records the lowest direct (36.0), indirect (35.2), and combined (35.6) attack success rates, with 82.5 benign success. One condition differs: baselines ran under the stock runner, Isaac under the Pokee harness. Efficiency, pricing, and portability Under the RULER workload on one B200-class GPU, TTFT is 23.6s at 1M and 72.9s at 10M. Prefill throughput rises with context, from 42,400 to 137,200 tokens/s, so a ten-fold longer prompt costs roughly three times the TTFT. List pricing is $0.15/$1.00 per million input/output tokens, marked provisional. Isaac also runs fully on-device on Intel Arc Pro B70 and Core Ultra Series 3 (Panther Lake), and on Qualcomm Snapdragon X2 Elite. Key Takeaways Pokee-Isaac 28B scores 93.3% on RULER at 10M tokens; every baseline in its panel returns 0.0 beyond 2M. Prefill reaches 137,200 tokens/s at 10M context on one B200; decode holds flat near 335 tokens/s. It leads BFCL v4 (70.94) and τ³-bench (0.662 avg), places second on Terminal-Bench 2.1, third on MCP-Atlas. Lowest combined attack success rate on DTAP (35.6) while keeping 82.5 benign task success. Weights are not published; deployment is licensed into VPC, on-premises, or on-device. Check out the Blog and Paper. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Pokee AI Releases Pokee-Isaac 28B: A 10M-Token Context Agentic Model Built to Run Inside the Customer Boundary appeared first on MarkTechPost.

Microsoft has open sourced code-testing-generator, a polyglot agent that writes unit tests and then proves they work. It ships in the dotnet-test plugin inside the MIT-licensed dotnet/skills repository. The agent targets a gap that coding assistants usually leave open. A prompt like ‘generate unit tests’ does not say which framework, file location or assertions to use. code-testing-generator settles those decisions by reading the repository before it writes anything. It then plans, writes, runs and checks the tests it produces. On Microsoft’s internal 152-task benchmark, it completed 140 tasks against 120 for stock GitHub Copilot. Both setups used the same model and prompts. Is it deployable Yes. It is an agent definition along with skills, not a hosted service, so it runs inside your existing coding agent and code stays local. Company stage: viable from solo maintainers upward. Startups and mid-market teams gain most, because the agent supplies repository research a small team has no time to encode. Enterprises can fork the language guidance to match internal frameworks. Industries: regulated or audit-heavy software estates — financial services, healthcare, insurance, public sector — plus platform teams paying down legacy test debt. Applications: backfilling tests on untested modules, generating tests for a pull-request diff, raising coverage before a release gate, and standardising conventions across polyglot monorepos. What the agent actually does It coordinates work through a Research-Plan-Implement (RPI) pipeline. It searches the repository for code needing tests, detects the language and test framework, reads existing tests for conventions, and finds the real build and test commands. That last step targets a specific failure: a test project that builds locally but never runs in CI because nothing registered it. The agent then picks one of three strategies. Direct writes and validates tests immediately. Single pass runs one cycle. Iterative repeats it for large scopes or coverage targets. It never modifies production code, and avoids tests that call external URLs, bind ports or depend on timing. The verification gate Before reporting completion, the agent runs five checks. It reasons about small code changes that should make the tests fail, a lightweight form of mutation testing. It looks for weak or missing assertions. It maps every requested scenario to a test. It builds the full workspace and runs the full suite. It confirms the repository’s own test command discovers the new tests. Benchmark results On Microsoft’s internal benchmark of 152 tasks from real repositories, the agent completed 140 (92.1%) versus 120 (78.9%) for stock GitHub Copilot on the same model and prompts (63% fewer failures). The gain is concentrated. On 89 vague prompts, the agent resolved 79 (88.8%) against 59 (66.3%), cutting failures from 30 to 10. On 63 detailed prompts, both scored 61 (96.8%). On 15 tasks targeting a specific diff, the agent passed all 15 and stock Copilot passed none. Notably, the agent generated 2.3% fewer tests (6,963 vs 7,129) at effectively identical line coverage (72.4% vs 72.2%). Average task time was 359 seconds against 380. Token use per completed task was 3.2% higher. On 45 .NET tasks, Claude Opus 4.8 reached 43/45 with the agent versus 35/45 stock; GPT-5.5 reached 41/45 versus 36/45. On the harder external SWE Atlas benchmark, completion was 16/44 versus 12/44. Explainer: how the agent turns one prompt into verified tests Key Takeaways Open source, MIT-licensed, polyglot unit-test agent from Microsoft’s .NET team. Research-Plan-Implement pipeline replaces one-shot generation with repository-aware planning. 92.1% vs 78.9% task completion against stock Copilot on the same model. Gains come almost entirely from vague prompts and diff-targeted requests. Fewer tests, same coverage, 5.5% faster — reliability, not volume. Check out the Technical details and Repo. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Microsoft Open Sources code-testing-generator: a Polyglot Unit-Test Agent That Hits 92.1% Task Completion Versus 78.9% for Stock Copilot appeared first on MarkTechPost.

arXiv:2608.05576v1 Announce Type: new Abstract: When a large language model (LLM) writes Harry Potter fanfiction, it reliably produces fundamental elements of the Hogwarts universe, such as recognizable places and characters. Human-written Harry Potter fanfictions, however, typically include these fundamentals and much more, incorporating stylistically irregular content and relationship-diverse plotlines. This gap between LLM and human writing has been noted across a variety of domains. LLMs tend to produce “average” writing, while human writing contains more diverse content that covers a broader distribution. Existing work has shown the existence of this distributional “gap”, but no work has proposed a systematic way to measure it. Our paper proposes a human-grounded framework that uses the empirical distribution of human writing on a topic to measure the distributional breadth of LLM-generated content on that same topic. We propose two metrics, LLM Coverage (LLM-Cov) and In-Boundary Rate (IBR), that separate the plausibility of LLM content from its distributional breadth. Across ideation and narrative tasks, we find that current LLMs produce plausible but narrow content that concentrates near the center of the human response space. Our framework can enable researchers to better assess the distributional breadth of LLM-authored content, which we term its “cultural reach”.

But cutting your runtime token burn is just the first problem.