Committee

On an afternoon in Seoul in March 2016, I watched a program I helped build put a stone on the fifth line of a Go board in what looked like a gift to its human opponent. Move 37 in game two of the five-game match looked so absurd that some commentators thought it was a programming glitch. It wasn’t. AlphaGo won the game, ultimately triumphing 4-1 over Lee Sedol, one of the greatest professional Go players of all time. “I thought AlphaGo was based on probability calculation and that it was merely a machine,” Lee said afterwards. “But when I saw this move, I changed my mind. Surely, AlphaGo is creative.”  When Deep Blue defeated then reigning world chess champion Garry Kasparov in 1997, it did so by looking six to eight moves ahead per player and evaluating 200 million chess positions per second, using rules hard-coded by humans. Go is a vastly more complex game. A stone’s worth depends on how distant groups and territory unfold over dozens of moves. Computing even a fraction of the possible outcomes would take a supercomputer billions of years. To win, AlphaGo had to sense who was ahead at a glance and even invent moves no human had thought to play. That is why many accounts of AlphaGo’s match against Lee portray move 37 as a flash of pure machine intuition. But that is a misunderstanding. It was actually AlphaGo’s powers of reasoning that made this creative choice—and these are powers that today’s AI lacks. If we want future AI systems to produce trustworthy results and really novel insights in fields like science and medicine, we need to equip them with genuine reasoning capabilities of this kind. AlphaGo is made up of two systems. The first, its policy network, was trained to guess what move a strong human would play. This “intuitive” part regarded move 37 as nothing special—a play that had a roughly one in 10,000 chance of being made by an expert human player. What made AlphaGo choose it was the program’s search machinery, which looked beyond immediate plausibility and weighed the future consequences of proposed moves. It explicitly constructed and searched a game tree with thousands of branches, each representing a different possible future.  A well-known theory in the behavioral sciences, popularized by Daniel Kahneman, distinguishes between two modes of human thought: System 1 is fast, gut-level, effortless; system 2, slow, step-by-step, and deliberative. AlphaGo offered a striking machine analogue of that split. Its networks supplied the hunches—this move looks promising, this position looks won—and its search supplied the deliberation, testing those hunches against the moves and countermoves that would follow. As in human cognition, neither half works alone. Intuition alone would never have opted for move 37, and brute-force search would have struggled to sieve through all the many possible moves. This is strikingly different from the way today’s AI models work. A large language model picks the next token, over and over. That amounts to system 1 in action—fast, associative, and surprisingly good pattern completion across almost every subject people write about. Not long after ChatGPT debuted, the field realized that language fluency alone falls short of true usefulness. The apparent solution was to make models that deliberate: Instead of answering immediately, they can now generate intermediate steps that decompose a problem, carry forward partial results, and influence subsequent reasoning—a process known as chain of thought. The gains have proved real, above all in mathematics and coding. But unlike AlphaGo’s search, this does not introduce a genuinely separate reasoning mechanism: The intermediate reasoning is still produced by the same next-token prediction process, iterated for longer before the model commits to an answer. Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize). First, these models typically maintain no explicit, persistent, and inspectable epistemic state. There is no open ledger that lays out the hypotheses a model is considering, the confidence it has in various explanations, the evidence it’s weighing, and the unresolved questions it’s holding onto—all things that should be systematically revised as new information arrives. Second, they lack a clean separation between what the system knows and how it manipulates that knowledge. Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs. Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another. This is a problem because in the high-stakes applications we all care about, such as medicine, engineering, and scientific research, it matters not only what a system concludes but also how it arrives at its conclusion. When mistakes happen—for example, in medical diagnosis and treatment—we need to be able to pinpoint what went wrong: Was the system’s reasoning at fault, did it draw on invalid evidence, or did it make incorrect assumptions?  This is why I recently left my position at Google DeepMind. I believe we need a fresh approach to machine reasoning—one that draws on AlphaGo’s architecture. AlphaGo maintains a record of what it knows about a given position: the game tree. This data structure contains all the variations, the possible futures, that AlphaGo has considered, each move and position being annotated with judgments made by its neural networks. As its reasoning progresses, AlphaGo updates the game tree and eventually synthesizes the information in it to decide which move to make.  Similarly, for general reasoning a system should maintain an epistemic state that represents what the system holds as settled, what it doubts, what it has ruled out, which questions stay open. Reasoning can then be understood as a sequence of moves that change the epistemic state to advance knowledge and reduce uncertainty: deducing consequences, breaking problems into parts, and—crucially—deciding what question to ask, calculation to perform, or experiment to run next. Of course, open-world reasoning is harder than playing a board

This week, I officially signed up for an unusual competition. One that rewards competitors for getting younger. I recently turned 40, and I don’t need reminding that both time and my chronological age only tick forward. But this game is focused on competitors’ biological ages—figures that are meant to provide a better way to measure the age-related health of our organs and bodies. Over a six-month period, around 500 of us will try to reverse our biological age as measured in a bunch of different ways. There’s even a leaderboard! The winners will include the person who shows the greatest difference between their chronological and biological age, as well as the person who manages to reverse their biological age the most. The competition is the brainchild of the neuroscientist and physician Christin Glorioso, who is also founder and CEO of NeuroAge Therapeutics. The company offers tools to track brain health and aging and is running a trial to find out if biological brain age can predict the onset of Alzheimer’s disease. Glorioso says she has several goals for the Younger contest. The first, she tells me, is to “explain to the world what aging clocks are.” Regular Checkup readers will be familiar with these tools, but for the uninitiated, they are basically designed to measure biological age. Many measure chemical signatures that form a layer on top of our DNA and seem to change with age. But others might involve looking at proteins or lipids in blood. Some are designed to measure functional health, while others predict when a person will die. There are more than a hundred clocks out there, and they probably each capture a specific aspect of the biological process of aging. These clocks are proving very useful for scientists, who use them to study aging and development in humans and many other species. But using them to estimate the biological age of an individual person is more controversial. They’re just not good enough for that yet. Part of the problem is that we don’t really understand what each clock is capturing. Glorioso hopes that data collected through the Younger contest might help answer that question. She’s hoping to generate “the world’s most comprehensive clock dataset.” Some longevity clinics try to get around the limits of biological age testing by using multiple clocks. That’s this competition’s approach too. As a standard competitor, I’ll be sending a blood spot sample to TruDiagnostic, a company that measures epigenetic chemical groups on DNA and offers to reveal an overall biological age as well as a person’s pace of aging and the biological ages of each of 11 organ systems, including the heart and brain. I’ll get a brain age score from NeuroAge once I complete the cognitive tasks on offer. I’ll also submit some physical measures like grip strength. And I’ll upload a series of selfies to an app developed by a team at Harvard that promises to tell me my face age. That’s exactly what it sounds like—an estimate of how old my face looks. I’m intrigued to see my scores, even though I know I shouldn’t put too much stock in them. The first time I took a test, four years ago, the company I used gave me a biological age that matched my chronological age, which was a little disappointing and, if I’m honest, a bit underwhelming. A year later, the same company told me I had a young heart but a relatively old brain, liver, and hormonal system. My lifestyle hasn’t changed much since then. Will the years have made much difference? (Ultra competitors—who pay $4,499 rather than the $999 cost of regular entry—are also offered a range of other blood tests and clocks, as well as MRI and DEXA scans. I’ve got complimentary press access to the standard package.) Glorioso is also hoping to gather information about the effects of some purported longevity approaches. During the competition, some participants will be offered red-light masks, access to a sauna, or other low-risk offerings that partner companies who are sponsoring the event to varying degrees might want to collect data on. “A lot of these companies just can’t afford to run [clinical] trials,” says Glorioso. She also sees the event as a public health initiative. A couple of years ago, David Sinclair, who studies aging at Harvard Medical School, and his colleagues estimated that if we could slow down aging in the general population and increase life expectancy by one year, it would save the US economy $38 trillion. The event officially kicks off in January, but around 120 people have already signed up, says Glorioso. Ultimately, she expects 500 participants. Each person’s six-month run starts as soon as they take their baseline measurements, whenever that may be. A handful of people have already started. “I think the majority of people are going to want to sign up after the holidays,” says Glorioso. “They’re not going to want to be trying to win an aging contest over Christmas and Hanukkah.” Glorioso took her own baseline measurements back in August. Those initial tests suggested that her biological age was around 10 years below her chronological age. That sounded pretty impressive to me until I looked at the leaderboard and saw that competitor @Fred had a biological age 28.8 years below his chronological age. The leaderboard is pretty sparse at the moment, as only seven people have logged their baseline measurements so far. Six of them have biological ages tracking low, but one 47-year-old has been given a biological age of 68.1. She’s competing publicly, so I can see the breakdown of her scores. The worst is for the speed at which she can stand from sitting in a chair—by that measure, she’s been given a biological age of 100. Hillary Lin, a longevity-focused physician in New York, has just started logging her baseline measurements for the competition. She tells me she’s taken biological age tests in the past and that they’ve tended to give scores below her chronological

Datalab has released OmniExtractBench, an open benchmark for structured document extraction. It tests how accurately a system fills a JSON schema from a PDF. The benchmark pools 620 documents from 4 existing benchmarks. One deterministic scorer grades all of them and explains each decision. The release lands while extraction vendors publish their own leaderboards. Datalab argues those leaderboards are hard to compare or audit. OmniExtractBench is its attempt at a shared yardstick. Is it deployable? Yes, the scorer installs from PyPI as omni-extract-bench (v0.1.7, Python 3.11+, SciPy only) under Apache 2.0. Rerunning vendors requires your own API keys and paid credits. What is OmniExtractBench? OmniExtractBench is a structured extraction benchmark built by Datalab. Each task gives a system a PDF and a JSON schema. The system returns JSON, which is scored value by value against a gold file. The code is on GitHub, and the data is on Hugging Face under CC BY 4.0. The 4 flaws it targets Datalab’s launch post names 4 recurring problems with existing extraction benchmarks: Bias: documents and scoring can favor the vendor that built the benchmark. Opaque harnesses: a low score may reflect a broken harness, not a weak model. Unclear scoring: readers cannot tell why a given document scored low. Narrow document variety: some suites hold only dense tables, others only clean, unscanned files. Where the 620 documents come from Suite Documents Upstream publisher Content ExtractBench 329 LlamaIndex Forms, filings, decks Internal 202 Datalab (synthetic) Dense scalar schemas, small documents LongExtractBench 47 micro1 (commissioned by Reducto) Very large tables LongArray-Extract 42 Extend Large tables with repeated scalars Regulatory filing forms are the largest category, at 88 documents. 128 documents are a single page. At the other end, 33 documents over 100 pages hold 40% of all pages. Datalab’s own synthetic suite is the second largest share. How the scorer works The scorer flattens prediction and gold JSON into addresses, which are paths to single values. It normalizes each value first, so “03/31/2024” matches “2024-03-31”. Tables are the hard part. Compared by position, one missed row shifts every row after it. We reran the scorer on a 100-row table missing its first row. Positional comparison scored 0%, while OmniExtractBench scored 99%. The fix is content-based pairing with the Hungarian algorithm. ExtractBench and LongArray-Extract already align rows this way. OmniExtractBench adds a verdict layer on top. 6 verdicts per value matched: paired, and the values agree. misread: paired, but the values differ. unfound: gold has a value, the prediction does not. fabricated: the schema allows it, gold is silent, the prediction fills it. invented_item: part of a predicted row that pairs with nothing. invented_field: an address the schema never declared. Accuracy is matched values over all verdicts. Precision divides matched values by predicted values. Recall divides them by gold values. The full rules are in the metric spec. The null rule Empty strings, None and whitespace count as omissions, so those addresses are dropped. Strings like “NA” or “-” remain real answers. This blocks a quiet exploit: padding a schema with empty optional fields to earn free matches. In our test, padded null fields added 0 verdicts. How it compares with other extraction benchmarks Benchmark Publisher Documents Row alignment Per-value explanation Scorer license Data license OmniExtractBench Datalab 620, from 4 sources Hungarian, by content Yes, 6 verdict types Apache 2.0 CC BY 4.0 ExtractBench LlamaIndex 370 Hungarian Per-field diffs, HTML report Apache 2.0 Apache 2.0 LongArray-Extract Extend 45, synthetic Hungarian Per-document score Not stated CC BY 4.0 LongExtractBench micro1 225, of which 50 public By row key Not stated MIT CC BY 4.0, labels only Sources linked on each benchmark name. Checked September 27, 2026. Who misses fields and who invents values Datalab scored 10 system configurations on the full corpus. Its accurate mode led at 93.85 accuracy. Datalab balanced (93.48) and Reducto deep_extract v2 (93.47) are effectively tied. Precision and recall then show how each system fails. Balanced: Datalab (both modes) and Reducto keep precision and recall within 0.6 points. Leans to misses: GPT 5.6-sol posts 95.11 precision but 84.99 recall. It loses 11.88% to unfound values. Gemini and Claude show the same pattern, less sharply. Leans to invented values: LlamaExtract has 93.13 recall but 86.57 precision, losing 9.03% to fabricated values. Extend loses 4.01% to invented items. Low on both: Mistral OCR 4.1 and Azure Content Understanding trail on both metrics, with recall lower still. How each system falls short of 100%, by verdict type. Source: Datalab. Run it yourself Copy CodeCopiedUse a different Browser uv pip install omni-extract-bench oeb score –pred pred.json –gt gold.json –schema schema.json –verdicts Install the [benchmark] extra and run oeb benchmark to rerun vendors. Runs are resumable, and each provider needs its own credentials. Datalab also suggests testing its playground on your own documents. Key Takeaways OmniExtractBench pools 620 documents from LlamaIndex, micro1, Extend and Datalab suites. A deterministic scorer gives every value 1 of 6 auditable verdicts. Content-based row pairing stops one missed row from zeroing a table. Dropping null addresses stops schema padding from inflating scores. Datalab accurate leads at 93.85; Datalab balanced and Reducto tie near 93.5. FAQ What does OmniExtractBench measure? It measures how accurately a system extracts values from a PDF into a JSON schema, scored per value. Is OmniExtractBench open source? Yes. The scorer is Apache 2.0 on GitHub and PyPI. The dataset is CC BY 4.0 on Hugging Face. Who built OmniExtractBench? Datalab built it. Datalab also maintains the open-source Marker and Surya document tools. Thanks to the Datalab team for the resources behind this article. Datalab supported and sponsored this content. The post Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks appeared first on MarkTechPost.

Cohere has released Embed 5, a new embedding model family. It targets enterprise search, RAG, and agentic retrieval. The model family ships in 2 tiers. Embed 5 Pro targets maximum retrieval quality. Embed 5 Fast targets latency and cost on the live query path. Both accept text, images, and fused text plus image inputs. Both cover 100+ languages and read up to 128K tokens. The key design choice: Pro and Fast share 1 embedding space. You can index with one and query with the other. Is it deployable today? Yes, both tiers are generally available on the Cohere API and Model Vault, Microsoft Foundry, and Amazon SageMaker. Private VPC or on-prem serving runs through vLLM. What Cohere Shipped The API model IDs are embed-v5.0-pro and embed-v5.0-fast, per Cohere’s model docs. Both output 2048, 1536, 1024, 768, 512, or 256 dimensions, with 2048 as default. Embeddings come back as float, int8, or binary. Pro costs $0.12 per 1M text tokens. Fast costs $0.08. Image inputs cost $0.40 per 1M tokens on both. Embed 5 can embed a page image directly. It can also fuse an image with its metadata into a single vector. That is important for scanned pages, slide decks, schematics, and charts, where text extraction drops information. Pro and Fast: One Index, Two Query Paths Cohere tested every corpus and query pairing across 40 development datasets. Normalized to Pro plus Pro at 100, a Pro index queried with Fast scored 98.4. An all-Fast setup scored 96.6. Cohere’s recommended pattern is to index with Pro and query with Fast. One constraint: both sides must use the same output dimension. The split targets agentic workloads. An agent may issue dozens of searches per task, and query latency compounds. Cohere team reports Fast processed 377.3 documents per second versus 159.7 for Pro. Benchmarks On ViDoRe V3, Embed 5 Pro averages 85.8, an 8.8-point gain over Embed 4. Fast averages 84.5. Voyage 4 Large scores 83.7, Gemini Embedding 2 scores 83.2, and OpenAI text-embedding-3-large scores 75.5. On Cohere’s parsed-PDF suite, Pro leads at 84.8 against Voyage 4 Large at 83.6. Finance is the strongest showing. Pro ranks first on FinanceBench (80.1), FinQA (90.0), and ViDoRe V3 Finance (85.0). Fast ranks second on all 3. Multilingual results are mixed. Pro leads the European-language average at 77. However, Gemini Embedding 2 beats Pro on 9 of 10 further languages in Cohere’s own results table. Those include Japanese, Arabic, Hindi, and Telugu. One important thing to note. Most numbers use RCP-nDCG@10, a new Cohere metric. It reorders a fixed candidate set, so it measures reranking quality more than first-stage retrieval. Cohere published the evaluation code, but independent replication is still pending. Storage Costs at Scale Embed 5 uses Matryoshka representation learning plus lower-precision outputs. A 2048-dim float32 vector needs 8 KB. A 1024-dim int8 vector needs 1 KB. A 256-dim binary vector needs 32 bytes. Across 100M chunks, raw storage drops from about 819 GB to 3.2 GB. Cohere recommends 1024-dim int8 as the default, citing near-full-precision quality. Interactive Explainer: How Embed 5 Works Cohere Embed 5 Explainer | Marktechpost Cohere Embed 5 · Interactive explainer1 / 6 1. What an embedding model actually does Text goes in, a vector of numbers comes out. Search then ranks documents by how close their vectors sit to the query vector. Press the button to watch it run. Query What happened to net interest margin last quarter? Embed and searchModel: embed-v5.0-pro · 1024 dims Documents ranked by cosine similarity Net interest margin narrowed 12 bps to 2.61% as deposit costs rose.0.00 Torque the mounting bolts to 45 Nm in a star pattern before refitting the cover.0.00 Employees accrue 1.5 days of paid leave for each month of service.0.00 Documents come from Cohere’s launch code sample. Vector bars and similarity scores here are illustrative, not real model output. 2. Two tiers, one embedding space Pro and Fast write vectors into the same space. Pick which model indexes the corpus and which one embeds the query. Quality is normalized so Pro plus Pro equals 100. Index corpus with ProFast Embed queries with ProFast Cohere’s recommended pattern: index with Pro, query with Fast. 0relative nDCG@10 Ring zoomed to the 90 to 100 range so small gaps are visible. Document throughput (docs per second, Cohere-measured) Embed 5 Fast377.3 Embed 5 Pro159.7 Price per 1M text tokens: Pro $0.12, Fast $0.08. Images cost $0.40 per 1M tokens on both tiers. 3. Shrink the vector, not the quality Matryoshka training lets you truncate dimensions. Lower precision outputs cut bytes further. Change the settings and watch storage move. Output dimensions 204815361024768512256 Precision float32int8binary Chunks in your index: 100,000,000 8,192 Bbytes per vector 819.2 GBraw vector storage 1xsmaller than 2048 float32 Storage vs 2048-dim float32 baseline100% Math: dims x bytes per value (4, 1, or 1/8). Cohere recommends 1024-dim int8 as the default sweet spot. Binary trades some accuracy and suits a first pass before reranking. 4. How it scores against rivals Cohere-reported averages using its new RCP-nDCG@10 metric. Treat these as vendor numbers until independent runs land. ViDoRe V3Parsed PDFsFinance Source: Cohere Embed 5 launch post, Sep 30 2026. RCP-nDCG@10 reorders a fixed candidate set, so it reflects reranking quality more than first-stage recall. 5. How much fits in one input A longer context means fewer chunks for long filings and manuals. Press play to compare maximum input length per call. Compare context windows Cohere Embed 5 Pro / Fast128K tokens Voyage 4 Large32,000 tokens Gemini Embedding 28,192 tokens OpenAI text-embedding-3-large8,191 tokens Limits taken from each vendor’s model docs. 128K is shown as 131,072 tokens for bar scale. 6. Is it deployable? Yes. Pick a route. Both tiers are generally available. Tap a route to see what it gives you. Cohere APIManaged endpoint Model VaultDedicated inference Microsoft FoundryAzure catalog Amazon SageMakerAWS Marketplace Private VPC / on-premServed with vLLM Cohere NorthBuilt-in retrieval ← Back Next → Built by MarktechpostData: Cohere blog · Cohere docs