Uncategorized

arXiv:2605.23440v4 Announce Type: replace Abstract: Joint Entity and Relation Extraction (JERE) is highly sensitive to training data quality, making data augmentation a natural way to improve generalization. However, existing augmentation methods often weaken entity relevance and disrupt semantic structure, limiting their effectiveness for JERE. In this paper, we propose textbf{Structured Semantic Data Augmentation (SSDAU)}, a method designed to preserve triple-aware semantic structure during augmentation. SSDAU segments text by entity labels, captures semantic features through context-aware encoding, and restructures entity semantics to generate augmented data. To distinguish semantically similar entities, SSDAU combines contextualized embeddings with traditional similarity scores. To reduce topic inconsistency, we apply BERTopic-based filtering to remove irrelevant augmentations. We evaluate SSDAU on datasets with different annotation types and compare its performance on five representative JERE models against seven popular augmentation baselines. Experiments show that SSDAU generates semantically consistent data, is more robust to ambiguity than non-LLM methods (8.95% vs. 23.58% average relative F1 decrease), and significantly outperforms strong alternatives in most settings.

The alert was raised on May 5. Four health-care workers in the Ituri Province of the Democratic Republic of the Congo had died from an unknown illness within four days. Rapid response teams were sent to investigate, and tests at a research center in Kinshasa revealed the culprit: the Bundibugyo virus, one of the viruses that cause Ebola. Suspected cases of the disease have snowballed in the last few weeks. By May 24, the WHO had estimated that 223 people had died from the disease. There were over 900 suspected cases. Today’s figures are likely to be higher. A couple of weeks ago, I covered the hantavirus outbreak aboard a cruise ship. Three people sadly died, but the outbreak itself was kept under control. There have been no further deaths, and passengers have been safely repatriated. The picture for Ebola is far bleaker. And there are several reasons why. The most obvious is the disease itself. Ebola is a severe disease with an average 50% fatality rate. Previous outbreaks have resulted in thousands of deaths. (Hantavirus also has a high fatality rate, but it doesn’t usually spread as easily between humans.)  Between 2014 and 2016, an Ebola outbreak in West Africa caused more than 11,000 deaths. A more recent outbreak, which took place between 2018 and 2020, caused 2,299 deaths before being brought under control with a vaccination campaign. But those outbreaks were caused by the Zaire virus, which has a different genetic sequence. There is no vaccine for the Bundibugyo virus. We don’t know if the two vaccines approved for Zaire might also work for Bundibugyo. There’s a concern they might even make things worse by interfering with a person’s immune response to the virus.   Scientists are working on potential Bundibugyo vaccines. But the most advanced efforts are still months away from clinical trials. There are no specific antiviral treatments for the virus, either. So to control the outbreak, health-care workers are trying to stop the spread of the disease. Ebolaviruses can be transmitted to humans by animals including fruit bats, chimpanzees, and gorillas. They can then spread between people via contact with bodily fluids such as blood or vomit. That’s why the virus is often spread among family members, to health-care workers, and during some burial services. The WHO advises isolating people who have the virus in treatment centers. It also recommends safe burial measures that limit physical contact with the deceased, for example. Communities need to be informed about the virus and how it spreads, and health professionals should be on hand to diagnose cases and track them. That’s all easier said than done in an era of misinformation. Some members of the community even doubt whether the disease is real. There have been three attacks on health-care facilities in the region in recent weeks. Last week, two treatment centers were burned down. The first incident occurred after relatives of a deceased man were prohibited from retrieving his (infectious) body. As a result of the second incident, 18 suspected cases reentered the community. A couple of days later, a group of men unleashed gunfire at Mongbwalu General Hospital, which was also treating people with Ebola. They were demanding the bodies of their deceased relatives. There are more causes for concern when it comes to the spread of the virus. The Ebola outbreak is thought to have originated in Mongbwalu, a high-traffic mining hub. People who caught the virus in Mongbwalu are thought to have sought care in neighboring districts. And the wider province borders both South Sudan and Uganda. So far, Uganda has reported seven confirmed cases and one death. South Sudan’s health ministry has said it will strengthen surveillance, but no cases have been reported in the country so far.  Violence in the region is making it much harder to contain the spread of the virus, too. Conflict involving multiple armed groups, including deadly attacks on civilians, has hampered humanitarian and health-care efforts. Poor infrastructure and damaged roads make matters even worse. Food insecurity is ravaging the region as well—this year, nearly 10 million people in the region face acute hunger. Together, these factors are making it “nearly impossible” to isolate people with Ebola and trace others who have been in contact with them, WHO director general Tedros Adhanom Ghebreyesus said in a statement earlier this week. The dismantling of US aid programs hasn’t helped either. US government funding for international health projects has steeply declined since the start of President Donald Trump’s second term. These cuts have harmed disease surveillance systems, according to the International Rescue Committee, a humanitarian nonprofit. “Funding cuts have left the region dangerously exposed,” Heather Reoch Kerr, the organization’s country director for the Democratic Republic of the Congo, said in a statement. “Years of underinvestment and recent funding cuts have left many health facilities without adequate protective equipment, surveillance capacity, or frontline support needed to respond quickly and safely.” The US has mobilized emergency funding for the outbreak, and a spokesperson for the State Department has argued that none of the administration’s actions have hampered the Ebola response. But health experts counter that the damage has already been done. On May 17, the WHO declared the Ebola outbreak a public health emergency of international concern. In a statement on Wednesday, Tedros described the situation as “a catastrophic collision of disease and conflict with the Ebola outbreak in Ituri province outpacing the response.”In an online appeal to residents on Wednesday, ahead of an in-person visit, Tedros pleaded for a ceasefire and commended the spirit of community members. He also acknowledged the steep challenges they face. “You are already carrying so much: malaria, hunger, insecurity, and the daily struggle to keep your families safe,” he wrote in French. “And now Ebola. It’s not fair, and I won’t pretend otherwise.” This article first appeared in The Checkup, MIT Technology Review’s weekly biotech newsletter. To receive it in your inbox every Thursday, and read articles like this first, sign up here.

Pope Leo XIV’s new encyclical on artificial intelligence includes a statement that warrants serious attention from technologists and policymakers: “Technology is never neutral.” Magnifica Humanitas (“Magnificent Humanity”) is a clarion call to all people to act with courage and solidarity as we enter an age already being transformed by artificial intelligence, the greatest change in human life since the Industrial Revolution. As the pope says, the choice before us—the choice AI presents—is one between the Tower of Babel and the rebuilding of our common humanity.  In the biblical story of the Tower of Babel, humans sought to build a massive structure that reached all the way to Heaven, only to have their project thwarted when God made those involved unable to understand one another. It was a pursuit fixated on relentless growth, divorced from any concern about God’s commandments or the human cost. It resulted in failure and atomization. The Book of Nehemiah, however, offers a contrasting narrative, in which the rebuilding of Jerusalem after a period of violence and displacement becomes an opportunity for humanity to show its collaborative resilience. As the encyclical puts it, “The city is reborn, not through the initiative of one man, but through the shared responsibility of all: men, women, priests, artisans, heads of households and young people all play a part. It is an undertaking with God at the center, which rebuilds relationships before rebuilding with stones.”  Is there any question which road we are currently barreling down? And can there be any doubt which we would do well to walk together?  We are both Catholics, members of religious communities and longtime advocates within the movement for socially responsible investment. Of particular interest to us and that movement is Pope Leo’s point that AI is not some force of nature or hyperrational, ineffable entity. Instead, he reminds us, AI is ultimately another commercial product, one emerging at a point in history when excessive power over commerce and the wider society has amassed in a vanishingly small number of hands.  It’s a powerful message. It’s also one that institutional investors have been acting on for years. This encyclical doesn’t break new ground so much as ratify a governance effort that’s already underway, led not by states or international bodies but by shareholders. When governments fail to meaningfully regulate, and corporations cannot be trusted to do what is beneficial beyond their own bottom line, people in society still have the power to set us on the right path, and indeed have the duty to do so.  Around the world, AI systems are being deployed at scale with remarkably little institutional oversight. There is no AI safety board. The US Federal Trade Commission has jurisdiction over unfair practices but limited authority over algorithmic design. The National Institute of Standards and Technology publishes guidance that most companies ignore. The EU AI Act is partially in force but addresses only a sliver of the deployment surface. Institutional investors have stepped into this vacuum. Coalitions including the membership of the Interfaith Center on Corporate Responsibility, representing investors managing over $400 billion in assets, have spent the past several proxy seasons filing resolutions demanding transparency, risk assessment, and accountability around AI deployment. Secular institutional investors have joined them, treating AI governance failures as material business risks. Shareholders have called tech giants including Alphabet, Amazon, Nvidia, Palantir, and Uber to account and demanded that AI not be used for acts of violence or other violations of human rights. The importance of this aspect of corporate governance was highlighted tragically in the opening hours of the war against Iran, when AI was used to help identify targets for thousands of missile strikes that killed hundreds of people.   Investors have also challenged executives at CVS and UnitedHealth Group to ensure that AI not be used to undermine the well-being of patients and quality of health care across the United States.  At companies including Meta and Microsoft, shareholders have decried the environmental impact of AI data centers, which consume vast amounts of energy and precious water resources, and in turn can emit large amounts of greenhouse gases.  Within creative industries, investors have challenged the leadership at companies like Disney, Netflix, and Warner Bros. to demand transparency about the ways they are using AI and to defend the inimitable human element in storytelling.  Soon, with OpenAI, Anthropic, and Grok all set to enter the public markets, we will be able to exert similar influence over what are now all privately held entities. These actions by concerned investors not only call out misdeeds but hold fast to an immutable truth: that it is wrong to use technology to kill, harm, or oppress people. Every human being has a right to safe and effective health care and the opportunity to earn a dignified living. The stories we tell each other matter and require the human creative spark.  Investor advocates hail from a range of faith traditions. Some have no formal religious faith. Yet in their informed and tenacious advocacy, all these people echo the calls embedded within Pope Leo’s encyclical and act on its declaration that “it is essential that the use of AI, especially when it touches on public goods and fundamental rights, be guided by clear criteria and effective oversight.”  Encyclicals mark time. A century from now, how will we be remembered for how we met this moment? Will we be seen as having been too timid or shortsighted to prevent a small group of unfathomably wealthy and self-interested people from seizing ever greater control over the human family’s shared destiny?  Or will the years ahead be remembered as a turning point that helped us rebuild our common humanity? Let this be a time when people of good will and diverse talents come together through their own magnificent humanity to build a future that honors our Creator. Father Séamus Finn, OMI, is a global leader in faith-based and socially responsible investing and a priest of the Oblates of Mary Immaculate, a missionary religious congregation.

GPU communication overhead is a measurable bottleneck in production AI workloads. According to data cited by the mKernel project, communication can consume 43.6% of the forward pass and 32% of end-to-end training time. Across popular Mixture-of-Experts (MoE) models, inter-device communication can account for up to 47% of total execution time. Researchers from UC Berkeley’s UCCL project have released mKernel, a library of persistent CUDA kernels that fuse intra-node NVLink communication, inter-node RDMA, and compute into a single kernel. The Problem: Host-Driven Communication The standard model for multi-GPU communication is host-driven: the CPU runs the control path and calls into a library like NCCL or NVSHMEM. The library issues the collective operation — an AllReduce, an AllGather, etc. — across GPUs. Compute and communication run on separate CUDA streams and overlap at kernel boundaries. The research team identifies two problems with this approach: (1) CPUs are not scaling with GPU compute. A GB300 NVL72 rack integrates 72 Blackwell Ultra GPUs and 36 Grace CPUs, delivering 720 PFLOP/s FP8/FP6, 1.44 EFLOP/s FP4 Tensor Core performance, and 130 TB/s of all-to-all intra-rack NVLink bandwidth. At those speeds, microsecond-scale host orchestration overhead — a cudaLaunchKernel call, a CPU-side “all writes done” check, an inter-stream event — shows up directly as pipeline bubbles. (2) Host-driven systems overlap compute and communication at coarse kernel boundaries. Finer-grained overlap at the tile or chunk level is not possible from the host side. The alternative is GPU-driven communication: the GPU itself triggers transfers, with communication fused into the same kernel as the compute. Most existing fused kernel libraries operate within a single node, or a single GPU. mKernel targets the multi-node case. What mKernel Does mKernel is a library of persistent CUDA kernels. Each kernel fuses intra-node NVLink communication, inter-node RDMA, and dense compute into a single kernel. Multi-GPU + multi-node, in one kernel: Both intra-node NVLink and inter-node RDMA live inside the same persistent kernel. Fine-grained intra-kernel overlap: Compute and communication overlap at tile/chunk granularity, covering both intra-node and inter-node GPU communication. Persistent kernel with SM specialization: CTAs self-assign roles: compute, intra-comm, inter-send, inter-reduce. The number of SMs dedicated to each role is tunable per shape. GPU-driven networking built on libibverbs: mKernel uses GPU-initiated RDMA writes without depending on NCCL or NVSHMEM. The communication backend is written from scratch to maximize performance and support heterogeneous networking devices. The Five Fused Kernels Kernel What it fuses Description AllGather + GEMM AllGather → GEMM Each rank holds a shard of A. While ranks gather peers’ shards over NVLink/RDMA, the local GEMM consumes tiles as soon as they arrive. GEMM + AllReduce GEMM → AllReduce Computes C = A @ B and reduces partial outputs across all ranks in one launch. Output tiles are pushed into the reduction tree the instant they’re produced. MoE Dispatch + GEMM All-to-All dispatch → grouped GEMM Routes MoE tokens to their expert ranks (intra-node NVLink + inter-node all-to-all) and runs the per-expert grouped GEMM in the same kernel. Tokens are processed as soon as they land — no staging buffer round-trip. Ring Attention Ring KV exchange → FlashAttention Sequence-parallel attention across ranks. Each step rotates a KV chunk around the ring while the local FlashAttention consumes the previously-received chunk. Compute and the ring send/recv run concurrently inside a single persistent kernel. GEMM + ReduceScatter GEMM → ReduceScatter Computes C = A @ B and reduce-scatters the output. Each output tile is reduced and forwarded to its owning rank as soon as it is produced. Evaluation Setup The research team evaluated mKernel on two 2-node × 8-H200 clusters that differ only in their inter-node fabric: Testbed Nodes × GPUs Intra-node Inter-node transport NIC AWS EFA 2 × 8 H200 NVLink AWS EFA / SRD 16 × 200 Gb/s EFA per node ConnectX-7 2 × 8 H200 NVLink InfiniBand 8 × 400 Gb/s NVIDIA ConnectX-7 per node mKernel was benchmarked against NCCL, Triton-distributed, Flux, Mercury, MagiAttention, Transformer-Engine, and ring-flash-attention. The team notes that further benchmarking at larger scale is still in progress. Backends and Requirements mKernel supports two networking backends: Backend Macro Transport Where it runs CX7 -DINTERNODE_BACKEND_IBVERBS libibverbs RC ConnectX-7 / InfiniBand / RoCE EFA -DINTERNODE_BACKEND_EFA libibverbs + efadv (SRD) AWS p5/p5e (H200, EFA) Both backends share the same host-side API and the same on-GPU kernel. Only the proxy/session implementation differs (session.h for CX7, session_efa.h for EFA). Requirements: NVIDIA Hopper GPUs (default build targets sm_90a), CUDA 12.9, Python with PyTorch. The CX7 backend requires libibverbs development headers and libraries. The EFA backend requires AWS EFA installation with libfabric, libibverbs, efadv, and EFA headers under EFA_HOME=/opt/amazon/efa by default. Marktechpost’s Visual Explainer UCCL mKernel — Multi-GPU, Multi-Node Fused Kernels Guide 01 / 07 — Overview What is mKernel? mKernel is an open-source library of persistent CUDA kernels from UC Berkeley’s UCCL project. It fuses intra-node NVLink communication, inter-node RDMA, and dense compute into a single kernel. Most existing fused kernel libraries operate within a single node or a single GPU. mKernel is designed from the start to span node boundaries. 43.6% of forward pass consumed by communication in production 47% of total execution time in popular MoE models 32% of end-to-end training time consumed by communication 02 / 07 — The Problem Why Host-Driven Communication Falls Short The standard model is host-driven: the CPU calls NCCL or NVSHMEM, which issues collective operations across GPUs. The UCCL team identifies two problems. CPUs are not scaling with GPUs. A GB300 NVL72 rack delivers 720 PFLOP/s FP8/FP6 and 1.44 EFLOP/s FP4. At those speeds, microsecond-scale overhead from cudaLaunchKernel, CPU-side sync checks, and inter-stream events shows up directly as pipeline bubbles. Overlap is too coarse. Host-driven systems overlap compute and communication only at kernel boundaries. Finer-grained overlap at the tile or chunk level is not possible from the host side. The answer: GPU-driven communication. The GPU itself triggers fine-grained transfers, fused into the same kernel as the compute. 03 / 07 — Design Four Core Design Properties 🖧 Multi-GPU + multi-node, in one kernel. Intra-node NVLink and inter-node RDMA both live

This year, there’s been a wave of notable energy companies going public via IPO in the US. The solar and battery company Solv Energy went public in February, to the tune of $6 billion. X-energy, which is building small modular nuclear reactors, did the same in April, and its stocks surged on its first day of trading to hit a $11.5 billion market cap. Most recently, the geothermal company Fervo Energy went public in mid-May, and its market cap is now about $12.4 billion. Those are all success stories in the IPO world. And it certainly doesn’t feel like a coincidence that all these companies are racing to provide electricity in an era of rising demand (partly due to data centers). Let’s take a look at how these firms are doing, what this moment says about the grid, and what’s coming next.  Let’s start with Fervo Energy, a company we’ve covered a lot over the years that’s working to develop enhanced geothermal energy. (We included it on our 2025 list of Climate Tech Companies to Watch.) While conventional geothermal requires finding specific spots with hot rock, water, and fractures to support a power plant, Fervo essentially uses fracking techniques to create the necessary conditions. The company was founded in 2017, and it raised about $1.5 billion from investors over the years before its IPO. Fervo’s first commercial project, Cape Station in Utah, is expected to have a capacity of about 500 megawatts. The first unit is set to start generating power for customers by October and the next two units by January 2027. The new funding from the IPO could help the company scale. Fervo currently has over 600 megawatts’ worth of binding power purchase agreements. And it has leases for land that could together generate more than 40 gigawatts of electricity. (As of 2024, the entire US geothermal fleet had a capacity of just 4 gigawatts.) The company also has an eye on cutting construction and drilling costs—its Cape Station plant is expected to cost about $7 per kilowatt, which is cheaper than new nuclear power plants but over twice the expense of building a new natural-gas plant in the US.  X-energy also aims to provide reliable clean power: it’s part of the wave of next-generation nuclear companies working on small modular reactors. The company is building high-temperature gas-cooled reactors, which flow helium over self-contained pebbles of nuclear fuel. These reactors will each generate 80 megawatts of electricity, less than one-tenth the output of larger ones like Unit 4 at Plant Vogtle in Georgia, the most recent addition to the commercial nuclear fleet in the US.   X-energy also saw its IPO go well, and prices surged in trading after the initial offering. One interesting tidbit here—the company had previously planned to go public in 2023 but decided against it because of difficult market conditions. The company is still years away from demonstrating its technology in a commercial project.  You may recall a story I wrote last year about its effort to build nuclear reactors at the site of a Dow Chemical plant in Texas. The company recently received a key environmental approval for that project, though it’s still waiting for the final green light from the Nuclear Regulatory Commission to start construction. Finally, Solv Energy builds solar and energy storage projects, mostly for utilities and independent power producers. Solar and batteries are some of the cheapest and easiest technologies to add to the grid, so this one could get a lot of capacity online, quickly. The company already has 21 gigawatts’ worth of projects operational across 35 states. Many companies in the energy sector are pinning their hopes on the rapid growth in data center construction and operation. The AI boom has transformed the energy landscape, pushing electricity demand higher in a country where it’s been relatively flat for the last decade or so. Solv Energy mentioned data centers over a dozen times in documents filed with the Securities and Exchange Commission before its IPO.  And Fervo and X-energy are particularly connected to the tech giants driving AI. Google has been a longtime investor in Fervo and also pioneered what it calls its clean transition tariff with the company. Amazon is a client of X-energy as well as an investor; it reportedly owns close to 20% of the company. Fervo and X-energy are also in industries that occupy a political sweet spot. President Trump and his administration have gone after wind power and other renewables, cutting off existing support and slowing approvals for new projects. Meanwhile, geothermal and particularly nuclear power have kept favor with the federal government and enjoyed continued tax credits and grant funding. If a few big leaders cash through these IPOs, it could help investors feel more confident about supporting the energy sector, even if that money is concentrated in later-stage ventures like these rather than earlier-stage companies.  We could see other firms, particularly in nuclear and geothermal, attempt a similar route in the year ahead. A key thing to watch here will be whether Fervo and X-energy in particular can succeed in scaling up and deploying their technology. If either of these companies stumbles or misses a timeline, it could have ripple effects for those hoping to follow in these very lucrative footsteps.  This article is from The Spark, MIT Technology Review’s weekly climate newsletter. To receive it in your inbox every Wednesday, sign up here. 

It is one thing to say AI will change the world. It is another to expect the class of 2026 to applaud it. In fact, when former Google CEO Eric Schmidt told University of Arizona graduates that their task is to help shape AI, he was met with a resounding chorus of boos. “I can hear you,” he said, before conceding that fears about disappearing jobs and a broken future were “rational.” This is not exactly the message one hopes to hear while sweating under a polyester gown and tallying student loan payments. Graduates have been jeering at AI pep talks at other commencements too, including ceremonies at the University of Central Florida and Middle Tennessee State University. Still, increasingly loud skepticism hasn’t stopped OpenAI from winning court cases, raising enormous sums of money, and launching new partnerships. And AI is even earning some unlikely cheerleaders: Reese Witherspoon has warned women to embrace it or be replaced by it.

Perplexity AI’s research team reimplemented their Unigram tokenizer from scratch in Rust and open-sourced the code in pplx-garden, their inference technology repository. At production input lengths, the new encoder cuts p50 latency by roughly 5x versus the Hugging Face tokenizers crate, ~2x versus SentencePiece (C++), and ~1.5x versus IREE’s tokenizer (C), with zero steady-state heap allocations. In production, it reduced CPU utilization in Perplexity’s inference stack by 5-6x and shaved double-digit milliseconds off reranker latency. Why Tokenization Became a Bottleneck LLM inference cost is typically framed around GPU work: KV caches, attention kernels, expert routing. But smaller models, such as embedding models, classifiers, and rerankers, tell a different story. These models are two to three orders of magnitude smaller than frontier transformers. A reranker scoring hundreds of candidate documents per request is a clear example. With a small model, GPU compute often finishes in single-digit milliseconds. Every input still passes through CPU-side tokenization first. When batch sizes are large, tokenization becomes a meaningful fraction of total request latency. Perplexity’s work targets XLM-RoBERTa, a model with a 250K-token Unigram vocabulary trained with SentencePiece. Fine-tuned RoBERTa-family encoders are a common production choice for ranking, retrieval, and similarity tasks. What is Unigram Tokenization? Unigram tokenization was introduced by Kudo in 2018 and is implemented in SentencePiece. It frames segmentation as a most-probable-path problem. Each vocabulary token has a learned log-probability. The tokenizer picks the segmentation whose token scores sum to the highest value. The algorithm used to find that best path is the Viterbi algorithm, a dynamic programming technique from 1967. Byte positions form graph layers and vocabulary tokens are edges spanning a contiguous byte range. The DP recurrence iterates over byte positions and updates the best-scoring path at each position. The outer loop runs in linear time relative to input length. The inner loop walks a vocabulary trie (a prefix tree structure) at each byte position. On a 16K-token input, this inner walk executes hundreds of thousands of trie transitions. It is the hot path. What was Slow in the Hugging Face Implementation The Hugging Face tokenizers crate is the default Rust tokenizer most teams reach for. Perplexity used it as the benchmark reference. At 514 tokens (512 + BOS/EOS injection), the reference implementation had three costly patterns: Bottleneck Mechanism Measured impact Allocation per match String::from_utf8 + AHashMap lookup per trie match 7,295 allocations at 514 tokens; 299,171 at 16K Pointer chase per byte AHashMap at every trie node; 4 dependent loads per byte step Dependent-load latency dominates the hot path L2 thrashing on long inputs DP table and output buffers freshly allocated each call L2 miss rate climbs from 8% at 128 tokens to 50% at 16K Per-token allocation is constant: roughly 2 KB and ~18 allocations per token, regardless of input size. The latency problem becomes severe at longer inputs when cumulative allocations overflow the per-core L2 cache. Establishing a Baseline Before Changing the Trie Before switching the trie structure, Perplexity first isolated how much cost came from unnecessary work alone. They made a zero-allocation port of the reference: same HashMap trie, but with a caller-owned scratch struct reused across calls and token IDs stored directly in trie nodes (removing the per-match string allocation and secondary hash-map lookup). This baseline already cut p50 latency to 155 µs at 514 tokens, down from 326 µs in the reference. Instructions retired dropped 2.4x. The remaining cost was the HashMap pointer chase itself, which the next step addressed. The Three Optimizations Optimization 1: Double-Array Trie The Hugging Face trie stores children in a HashMap at every node. Each byte step requires a hash computation, two pointer dereferences, and a heap access. Perplexity replaced this with a double-array trie, the same structure used by SentencePiece and IREE, originally introduced by Aoe in 1989. A double-array trie encodes the entire trie in two flat integer arrays, base and check. A child lookup is: next = base[node] + byte, then verify check[next] == node. That is two array reads, one integer add, and one comparison, with no hashing and no pointer chasing. For XLM-RoBERTa’s 250K vocab, the whole trie fits in ~9 MB of contiguous memory. The hot working set per encode is on the order of 100 KB, which fits in L2 cache. Unlike SentencePiece and IREE, which are general-purpose libraries with lattice bookkeeping and multi-stage pipelines, Perplexity inlined the trie directly in the Viterbi loop and dropped that overhead entirely. Result at 514 tokens: p50 dropped from 155 µs (zero-allocation baseline) to 68 µs. Wall-clock fell 4.8x from the original reference. Optimization 2: Bitmap and Inline Packing The double-array trie still requires two dependent array loads per byte step: first the parent’s base offset, then the check array to confirm the transition is valid. Perplexity replaced the check array with a per-node bitmap (four 64-bit words, 32 bytes) that records which of the 256 possible bytes have valid child transitions. A bitmap lookup compiles to a single bit test against one 64-bit word. The check array is used only during trie construction and dropped from the runtime layout entirely. They also packed all four per-node fields (bitmap, base, token ID, and score) into a single 64-byte cache line, matching CPU cache line width exactly. One trie step now loads a single cache line covering the bitmap for the next-byte check, the base offset for the child slot, and the token ID and score at terminal nodes. Trade-off: trie size grows from ~9 MB to ~50 MB (780K nodes x 64 bytes). The hot working set per encode remains ~100 KB. Result at 514 tokens: Additional 4.5% wall-clock reduction. L2 accesses dropped from 4.6K to 1.8K per encode. Optimization 3: Huge Pages for the Trie At 50 MB, the trie spans roughly 12,000 virtual pages on a default Linux system using 4 KB pages. The first-level data TLB on Intel Sapphire Rapids holds 96 entries. Each Viterbi step touches a different trie node, so TLB misses accumulate. Over a 512-token encode, Perplexity estimated roughly