Committee

Datalab has released OmniExtractBench, an open benchmark for structured document extraction. It tests how accurately a system fills a JSON schema from a PDF. The benchmark pools 620 documents from 4 existing benchmarks. One deterministic scorer grades all of them and explains each decision. The release lands while extraction vendors publish their own leaderboards. Datalab argues those leaderboards are hard to compare or audit. OmniExtractBench is its attempt at a shared yardstick. Is it deployable? Yes, the scorer installs from PyPI as omni-extract-bench (v0.1.7, Python 3.11+, SciPy only) under Apache 2.0. Rerunning vendors requires your own API keys and paid credits. What is OmniExtractBench? OmniExtractBench is a structured extraction benchmark built by Datalab. Each task gives a system a PDF and a JSON schema. The system returns JSON, which is scored value by value against a gold file. The code is on GitHub, and the data is on Hugging Face under CC BY 4.0. The 4 flaws it targets Datalab’s launch post names 4 recurring problems with existing extraction benchmarks: Bias: documents and scoring can favor the vendor that built the benchmark. Opaque harnesses: a low score may reflect a broken harness, not a weak model. Unclear scoring: readers cannot tell why a given document scored low. Narrow document variety: some suites hold only dense tables, others only clean, unscanned files. Where the 620 documents come from Suite Documents Upstream publisher Content ExtractBench 329 LlamaIndex Forms, filings, decks Internal 202 Datalab (synthetic) Dense scalar schemas, small documents LongExtractBench 47 micro1 (commissioned by Reducto) Very large tables LongArray-Extract 42 Extend Large tables with repeated scalars Regulatory filing forms are the largest category, at 88 documents. 128 documents are a single page. At the other end, 33 documents over 100 pages hold 40% of all pages. Datalab’s own synthetic suite is the second largest share. How the scorer works The scorer flattens prediction and gold JSON into addresses, which are paths to single values. It normalizes each value first, so “03/31/2024” matches “2024-03-31”. Tables are the hard part. Compared by position, one missed row shifts every row after it. We reran the scorer on a 100-row table missing its first row. Positional comparison scored 0%, while OmniExtractBench scored 99%. The fix is content-based pairing with the Hungarian algorithm. ExtractBench and LongArray-Extract already align rows this way. OmniExtractBench adds a verdict layer on top. 6 verdicts per value matched: paired, and the values agree. misread: paired, but the values differ. unfound: gold has a value, the prediction does not. fabricated: the schema allows it, gold is silent, the prediction fills it. invented_item: part of a predicted row that pairs with nothing. invented_field: an address the schema never declared. Accuracy is matched values over all verdicts. Precision divides matched values by predicted values. Recall divides them by gold values. The full rules are in the metric spec. The null rule Empty strings, None and whitespace count as omissions, so those addresses are dropped. Strings like “NA” or “-” remain real answers. This blocks a quiet exploit: padding a schema with empty optional fields to earn free matches. In our test, padded null fields added 0 verdicts. How it compares with other extraction benchmarks Benchmark Publisher Documents Row alignment Per-value explanation Scorer license Data license OmniExtractBench Datalab 620, from 4 sources Hungarian, by content Yes, 6 verdict types Apache 2.0 CC BY 4.0 ExtractBench LlamaIndex 370 Hungarian Per-field diffs, HTML report Apache 2.0 Apache 2.0 LongArray-Extract Extend 45, synthetic Hungarian Per-document score Not stated CC BY 4.0 LongExtractBench micro1 225, of which 50 public By row key Not stated MIT CC BY 4.0, labels only Sources linked on each benchmark name. Checked September 27, 2026. Who misses fields and who invents values Datalab scored 10 system configurations on the full corpus. Its accurate mode led at 93.85 accuracy. Datalab balanced (93.48) and Reducto deep_extract v2 (93.47) are effectively tied. Precision and recall then show how each system fails. Balanced: Datalab (both modes) and Reducto keep precision and recall within 0.6 points. Leans to misses: GPT 5.6-sol posts 95.11 precision but 84.99 recall. It loses 11.88% to unfound values. Gemini and Claude show the same pattern, less sharply. Leans to invented values: LlamaExtract has 93.13 recall but 86.57 precision, losing 9.03% to fabricated values. Extend loses 4.01% to invented items. Low on both: Mistral OCR 4.1 and Azure Content Understanding trail on both metrics, with recall lower still. How each system falls short of 100%, by verdict type. Source: Datalab. Run it yourself Copy CodeCopiedUse a different Browser uv pip install omni-extract-bench oeb score –pred pred.json –gt gold.json –schema schema.json –verdicts Install the [benchmark] extra and run oeb benchmark to rerun vendors. Runs are resumable, and each provider needs its own credentials. Datalab also suggests testing its playground on your own documents. Key Takeaways OmniExtractBench pools 620 documents from LlamaIndex, micro1, Extend and Datalab suites. A deterministic scorer gives every value 1 of 6 auditable verdicts. Content-based row pairing stops one missed row from zeroing a table. Dropping null addresses stops schema padding from inflating scores. Datalab accurate leads at 93.85; Datalab balanced and Reducto tie near 93.5. FAQ What does OmniExtractBench measure? It measures how accurately a system extracts values from a PDF into a JSON schema, scored per value. Is OmniExtractBench open source? Yes. The scorer is Apache 2.0 on GitHub and PyPI. The dataset is CC BY 4.0 on Hugging Face. Who built OmniExtractBench? Datalab built it. Datalab also maintains the open-source Marker and Surya document tools. Thanks to the Datalab team for the resources behind this article. Datalab supported and sponsored this content. The post Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks appeared first on MarkTechPost.

Cohere has released Embed 5, a new embedding model family. It targets enterprise search, RAG, and agentic retrieval. The model family ships in 2 tiers. Embed 5 Pro targets maximum retrieval quality. Embed 5 Fast targets latency and cost on the live query path. Both accept text, images, and fused text plus image inputs. Both cover 100+ languages and read up to 128K tokens. The key design choice: Pro and Fast share 1 embedding space. You can index with one and query with the other. Is it deployable today? Yes, both tiers are generally available on the Cohere API and Model Vault, Microsoft Foundry, and Amazon SageMaker. Private VPC or on-prem serving runs through vLLM. What Cohere Shipped The API model IDs are embed-v5.0-pro and embed-v5.0-fast, per Cohere’s model docs. Both output 2048, 1536, 1024, 768, 512, or 256 dimensions, with 2048 as default. Embeddings come back as float, int8, or binary. Pro costs $0.12 per 1M text tokens. Fast costs $0.08. Image inputs cost $0.40 per 1M tokens on both. Embed 5 can embed a page image directly. It can also fuse an image with its metadata into a single vector. That is important for scanned pages, slide decks, schematics, and charts, where text extraction drops information. Pro and Fast: One Index, Two Query Paths Cohere tested every corpus and query pairing across 40 development datasets. Normalized to Pro plus Pro at 100, a Pro index queried with Fast scored 98.4. An all-Fast setup scored 96.6. Cohere’s recommended pattern is to index with Pro and query with Fast. One constraint: both sides must use the same output dimension. The split targets agentic workloads. An agent may issue dozens of searches per task, and query latency compounds. Cohere team reports Fast processed 377.3 documents per second versus 159.7 for Pro. Benchmarks On ViDoRe V3, Embed 5 Pro averages 85.8, an 8.8-point gain over Embed 4. Fast averages 84.5. Voyage 4 Large scores 83.7, Gemini Embedding 2 scores 83.2, and OpenAI text-embedding-3-large scores 75.5. On Cohere’s parsed-PDF suite, Pro leads at 84.8 against Voyage 4 Large at 83.6. Finance is the strongest showing. Pro ranks first on FinanceBench (80.1), FinQA (90.0), and ViDoRe V3 Finance (85.0). Fast ranks second on all 3. Multilingual results are mixed. Pro leads the European-language average at 77. However, Gemini Embedding 2 beats Pro on 9 of 10 further languages in Cohere’s own results table. Those include Japanese, Arabic, Hindi, and Telugu. One important thing to note. Most numbers use RCP-nDCG@10, a new Cohere metric. It reorders a fixed candidate set, so it measures reranking quality more than first-stage retrieval. Cohere published the evaluation code, but independent replication is still pending. Storage Costs at Scale Embed 5 uses Matryoshka representation learning plus lower-precision outputs. A 2048-dim float32 vector needs 8 KB. A 1024-dim int8 vector needs 1 KB. A 256-dim binary vector needs 32 bytes. Across 100M chunks, raw storage drops from about 819 GB to 3.2 GB. Cohere recommends 1024-dim int8 as the default, citing near-full-precision quality. Interactive Explainer: How Embed 5 Works Cohere Embed 5 Explainer | Marktechpost Cohere Embed 5 · Interactive explainer1 / 6 1. What an embedding model actually does Text goes in, a vector of numbers comes out. Search then ranks documents by how close their vectors sit to the query vector. Press the button to watch it run. Query What happened to net interest margin last quarter? Embed and searchModel: embed-v5.0-pro · 1024 dims Documents ranked by cosine similarity Net interest margin narrowed 12 bps to 2.61% as deposit costs rose.0.00 Torque the mounting bolts to 45 Nm in a star pattern before refitting the cover.0.00 Employees accrue 1.5 days of paid leave for each month of service.0.00 Documents come from Cohere’s launch code sample. Vector bars and similarity scores here are illustrative, not real model output. 2. Two tiers, one embedding space Pro and Fast write vectors into the same space. Pick which model indexes the corpus and which one embeds the query. Quality is normalized so Pro plus Pro equals 100. Index corpus with ProFast Embed queries with ProFast Cohere’s recommended pattern: index with Pro, query with Fast. 0relative nDCG@10 Ring zoomed to the 90 to 100 range so small gaps are visible. Document throughput (docs per second, Cohere-measured) Embed 5 Fast377.3 Embed 5 Pro159.7 Price per 1M text tokens: Pro $0.12, Fast $0.08. Images cost $0.40 per 1M tokens on both tiers. 3. Shrink the vector, not the quality Matryoshka training lets you truncate dimensions. Lower precision outputs cut bytes further. Change the settings and watch storage move. Output dimensions 204815361024768512256 Precision float32int8binary Chunks in your index: 100,000,000 8,192 Bbytes per vector 819.2 GBraw vector storage 1xsmaller than 2048 float32 Storage vs 2048-dim float32 baseline100% Math: dims x bytes per value (4, 1, or 1/8). Cohere recommends 1024-dim int8 as the default sweet spot. Binary trades some accuracy and suits a first pass before reranking. 4. How it scores against rivals Cohere-reported averages using its new RCP-nDCG@10 metric. Treat these as vendor numbers until independent runs land. ViDoRe V3Parsed PDFsFinance Source: Cohere Embed 5 launch post, Sep 30 2026. RCP-nDCG@10 reorders a fixed candidate set, so it reflects reranking quality more than first-stage recall. 5. How much fits in one input A longer context means fewer chunks for long filings and manuals. Press play to compare maximum input length per call. Compare context windows Cohere Embed 5 Pro / Fast128K tokens Voyage 4 Large32,000 tokens Gemini Embedding 28,192 tokens OpenAI text-embedding-3-large8,191 tokens Limits taken from each vendor’s model docs. 128K is shown as 131,072 tokens for bar scale. 6. Is it deployable? Yes. Pick a route. Both tiers are generally available. Tap a route to see what it gives you. Cohere APIManaged endpoint Model VaultDedicated inference Microsoft FoundryAzure catalog Amazon SageMakerAWS Marketplace Private VPC / on-premServed with vLLM Cohere NorthBuilt-in retrieval ← Back Next → Built by MarktechpostData: Cohere blog · Cohere docs

In this tutorial, we implement Kauldron, the JAX training library from Google Research that describes itself as optimized for research velocity and modularity, and we take those two words literally by testing what they actually buy us. We install it, then spend the first half of the notebook on the three mechanisms that make Kauldron different from a stack of Flax and Optax: konfig, which turns an experiment into a tree of plain dictionaries that round-trip through JSON; kontext, which wires components together with string key paths so a loss never imports the model it scores; and the runtime shape checker, whose named axes bind across arguments and report what they were bound to when something does not match. We then write a custom loss and a custom metric in the shape the framework expects, train a real Trainer on synthetic in-memory data with no download and no accelerator, and monitor an inner layer of the model without editing the model. We finish by running a five-variant sweep in which every experiment differs by one config line, and by letting a training run checkpoint itself and resume where it stopped. Copy CodeCopiedUse a different Browser import os import sys import json import textwrap import traceback import subprocess RESULTS = {} def banner(title): print(“n” + “=” * 78) print(title) print(“=” * 78) def section(name): def wrap(fn): def run(*a, **kw): banner(name) try: out = fn(*a, **kw) RESULTS[name] = out if isinstance(out, str) else “ok” return out except Exception as e: RESULTS[name] = f”SKIPPED / FAILED -> {type(e).__name__}: {e}” print(f”n[!] {name} did not complete: {type(e).__name__}: {e}”) traceback.print_exc(limit=3) return None return run return wrap banner(“0. Install Kauldron, and the one compatibility patch you need today”) subprocess.run([sys.executable, “-m”, “pip”, “install”, “-q”, “kauldron==1.4.2”], check=True) import jax from etils.enp import array_spec as _array_spec # jax >= 0.10.1 moved `jax._src.prng`, but etils <= 1.14.0 still reaches for it whenever it # inspects an array’s dtype. Kauldron calls that code on every batch, so without this two-line # patch a Trainer raises AttributeError before it finishes a single step. The replacement uses # jax’s own public dtype API and is a no-op on older jax. if not hasattr(jax._src, “prng”): _array_spec._is_jax_random_dtype = lambda dt: jax.dtypes.issubdtype(dt, jax.dtypes.prng_key) import numpy as np import optax import flax from flax import linen as nn import kauldron from kauldron import kd, konfig, kontext from kauldron.typing import Float, typechecked print(f” kauldron {kauldron.__version__} | jax {jax.__version__} | flax {flax.__version__}” f” | optax {optax.__version__}”) print(f” devices: {jax.devices()}”) print(“n Kauldron’s pitch is modularity: it is the glue, not the framework. Four pieces do the work:”) print(” konfig -> your experiment IS a Python call tree, and that tree is a plain dict”) print(” kontext -> parts are wired by string key paths, so they never import each other”) print(” ktyping -> Float[‘*b h w c’] checked at runtime, with named axes bound across args”) print(” kd.train -> Trainer: model + data + losses + metrics + optimizer, and nothing else”) print(“n Everything below runs on a CPU runtime with no dataset download: the data is synthetic.”) We install Kauldron and apply the one compatibility patch the current release combination needs. jax 0.10.1 moved the private module jax._src.prng, and etils up to 1.14.0 still reaches for it whenever it inspects an array’s dtype, which is a code path Kauldron runs on every batch. Without the two-line replacement below, which uses jax’s own public dtype API and is a no-op on older versions, a Trainer raises AttributeError before it completes a single step. With it in place we import the four pieces that do the work: konfig for the config system, kontext for the wiring, the typing module for runtime shape checks, and kd.train for the Trainer itself. Everything afterwards runs on a CPU runtime, because the only dataset in this notebook is one we generate. Copy CodeCopiedUse a different Browser @section(“1. A config is a call tree, and a call tree is a dict”) def config_is_a_dict(): with konfig.imports(): import optax as coptax # looks like optax, builds ConfigDict instead cfg = coptax.adam(learning_rate=0.003) print(f” cfg = {cfg}”) print(f” type = {type(cfg).__name__}”) print(f” __qualname__ = {cfg.__qualname__!r} <- the call, stored as data”) cfg.learning_rate = 1e-4 # configs are mutable optimizer = konfig.resolve(cfg) # …until you resolve them print(f” after cfg.learning_rate = 1e-4 -> resolve() gives {type(optimizer).__name__}”) print(“n An arbitrarily complex optimizer is still just nested dicts:”) chain = coptax.chain( coptax.clip_by_global_norm(1.0), coptax.scale_by_adam(b2=0.99), coptax.scale_by_learning_rate(0.003), ) as_json = json.dumps(json.loads(chain.to_json()), indent=2) print(textwrap.indent(as_json, ” “)) rebuilt = konfig.resolve(konfig.ConfigDict(json.loads(chain.to_json()))) print(f” JSON -> ConfigDict -> resolve() -> {type(rebuilt).__name__}”) print(” optax has no idea konfig exists. No base class, no registry, no decorator.”) return f”optax.chain -> JSON -> {type(rebuilt).__name__}” config_is_a_dict() We start with konfig, because it is the piece the rest of the library is built on. Inside a konfig.imports() block, importing optax gives us something that looks and autocompletes like optax but builds configuration instead of objects, so optax.adam(learning_rate=0.003) returns a ConfigDict holding the qualified name of the call and its arguments rather than an optimizer. That config is mutable until konfig.resolve turns it into the real thing, and because it is only nested dictionaries, an arbitrarily complex optax.chain serialises to JSON and comes back as a working optimizer. The important part is what optax had to do to support this: nothing. There is no base class, no registry, and no decorator anywhere in optax, and the same applies to any library we configure this way. Copy CodeCopiedUse a different Browser @section(“2. cfg.ref: change one number, everything downstream follows”) def config_references(): with konfig.imports(): import optax as coptax from kauldron import kd as ckd cfg = ckd.train.Trainer() cfg.num_train_steps = 1000 cfg.schedules = { “lr”: coptax.warmup_cosine_decay_schedule( init_value=0.0, peak_value=1e-3, warmup_steps=100, decay_steps=cfg.ref.num_train_steps, # <- a reference, not the value 1000 ) } at_1000 = konfig.resolve(cfg.schedules[“lr”]) cfg.num_train_steps = 200 # one edit… at_200 = konfig.resolve(cfg.schedules[“lr”]) # …and the schedule already knows print(f” {‘progress’:>10s} {‘lr @ 1000 steps’:>18s} {‘lr @ 200 steps’:>16s}”) for frac in (0.1, 0.5, 0.9): print(f” {frac:>9.0%} {float(at_1000(int(1000*frac))):>18.6f}” f” {float(at_200(int(200*frac))):>16.6f}”) print(“n Without .ref the schedule would have frozen 1000 into itself,

Cloudflare has released Clef and Clef-flash, the first models trained by its Workers AI team. They are decision models, not chatbots. Each reads an input state and a schema of typed questions. It returns a probability for every allowed answer, with no free-form text. Both are open-weight under Apache 2.0 and compatible with TypeSafe AI’s Jev API. Is it deployable? Yes, Both models run today on Workers AI, and the weights are on Hugging Face for self-hosting. What a Decision Model Does An LLM generates tokens one at a time, and its output still needs parsing. A decision model only answers a fixed set of questions about an input. Clef supports 3 question types: noul: yes/no, returns the probability of yes. choice: picks 1 named option, with per-option probabilities and a confidence value. score: rates against an ordered rubric, returning a probability-weighted score. On Workers AI, 1 request carries up to 64 questions and up to 4 images. TypeSafe AI launched Jev, its first ‘System One’ model, on September 15, 2026. Open alternatives like Kev-9B and Laya followed. Clef uses the same System One API. Switching from Jev means changing the endpoint and model name. How Clef Works Clef is post-trained from Qwen3.8-27B, and Clef-flash from Qwen3.5-9B. Both keep the backbone’s vision encoder. Inference has 2 stages. The backbone first runs a single prefill-only pass over the state and questions. A small transformer, the joint schema head, then reads the final hidden states. It routes evidence to each question, lets fields cross-attend, and scores all options jointly. A per-question softmax turns logits into probabilities. Training froze both backbones and jointly optimized the routing head with rank-256 low-rank adapters. The loss pairs label-smoothed cross-entropy with a Brier loss for calibration. A secondary objective, Reinforcement Learning for Calibrated Decisions (RLCD), gives partial credit to adjacent ordinal choices. Interactive Explainer Benchmarks: Where Clef Wins and Where It Does Not On Cloudflare’s 10-benchmark shortlist from the Decision Index 0.2.1 suite, a Clef model scored highest on 7. BANKING77 (macro-F1): Clef 94.20 vs Jev 79.74. CLINC150+OOS (macro-F1): Clef 97.43 vs Jev 89.27. Home appliances (case exact): Clef-flash 97.73 vs Jev 52.27. Jev keeps clear leads elsewhere. The full model card shows Jev ahead on GPQA Diamond (78.3 vs 48.0). It also leads MMLU-Pro (82.7 vs 65.9) and BBH (92.9 vs 73.7). On TypeSafe’s own workflow evals, Clef beat Jev in 3 of 4 areas, by small margins. Invoice processing was 64.7 vs 61.8, customer service 76.3 vs 76.0, and security incidents 62.9 vs 61.7. Jev leads agent trace observability, 71.6 vs 68.5. In Cloudflare’s threat intelligence workflow, Clef classified a domain in 2.2 seconds. gpt-oss-120b took 4.7 seconds. All numbers are vendor-reported, with no independent replication yet. Feature Comparison Feature Clef Clef-flash Jev Kev-9B Laya Developer Cloudflare Cloudflare TypeSafe AI Jared Palmer Convai Innovations Size 27B 9B Not disclosed 9B + 45.4M LoRA 421M Backbone Qwen3.8-27B Qwen3.5-9B Not disclosed Qwen3.5-9B-Base ModernBERT-large Weights Apache 2.0 Apache 2.0 Hosted API Apache 2.0 Apache 2.0 Image input Yes Yes No No No Context 65,536 65,536 32K (per Cloudflare) 65,536 (8,192 validated) 512 (English) Median latency* 209.3 ms 38.8 ms 524.1 ms 51.4 ms 5.8 ms Hosted price (input) $0.24/M $0.09/M $0.042/M Self-host Self-host *Cloudflare’s internal Decision Index run. All 5 implement the System One API. Sources: Clef docs, Clef-flash docs, Clef card, Jev post, Kev-9B card, Laya card. Deployment and Fine-Tuning Both models are callable through the Workers AI binding (env.AI.run()), the REST API, or AI Gateway. For self-hosting, the model cards list testing on a single H200 with BF16 weights. Cloudflare also announced a reinforcement learning service for tuning Clef on private data. It starts with Cloudflare’s forward-deployed engineers, with a self-serve platform later. The pipeline combines AI Gateway, Workers AI, Containers and a new Trainer component. Teams can apply via the design partner form. Key Takeaways Clef (27B) and Clef-flash (9B) are Apache 2.0, Jev-compatible decision models. Median latency: 209.3 ms for Clef, 38.8 ms for Clef-flash, 524.1 ms for Jev. Clef reads text, JSON, images and video within a 64K-token context window. Jev still leads on knowledge-heavy tests like GPQA Diamond, MMLU-Pro and BBH. An RL fine-tuning service starts with Cloudflare’s forward-deployed engineers. Check out the Model weight, Demo and Technical details. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Cloudflare Releases Clef and Clef-flash: Open-Weight Decision Models That Return Typed Probabilities Instead of Text appeared first on MarkTechPost.