Committee

GPU communication overhead is a measurable bottleneck in production AI workloads. According to data cited by the mKernel project, communication can consume 43.6% of the forward pass and 32% of end-to-end training time. Across popular Mixture-of-Experts (MoE) models, inter-device communication can account for up to 47% of total execution time. Researchers from UC Berkeley’s UCCL project have released mKernel, a library of persistent CUDA kernels that fuse intra-node NVLink communication, inter-node RDMA, and compute into a single kernel. The Problem: Host-Driven Communication The standard model for multi-GPU communication is host-driven: the CPU runs the control path and calls into a library like NCCL or NVSHMEM. The library issues the collective operation — an AllReduce, an AllGather, etc. — across GPUs. Compute and communication run on separate CUDA streams and overlap at kernel boundaries. The research team identifies two problems with this approach: (1) CPUs are not scaling with GPU compute. A GB300 NVL72 rack integrates 72 Blackwell Ultra GPUs and 36 Grace CPUs, delivering 720 PFLOP/s FP8/FP6, 1.44 EFLOP/s FP4 Tensor Core performance, and 130 TB/s of all-to-all intra-rack NVLink bandwidth. At those speeds, microsecond-scale host orchestration overhead — a cudaLaunchKernel call, a CPU-side “all writes done” check, an inter-stream event — shows up directly as pipeline bubbles. (2) Host-driven systems overlap compute and communication at coarse kernel boundaries. Finer-grained overlap at the tile or chunk level is not possible from the host side. The alternative is GPU-driven communication: the GPU itself triggers transfers, with communication fused into the same kernel as the compute. Most existing fused kernel libraries operate within a single node, or a single GPU. mKernel targets the multi-node case. What mKernel Does mKernel is a library of persistent CUDA kernels. Each kernel fuses intra-node NVLink communication, inter-node RDMA, and dense compute into a single kernel. Multi-GPU + multi-node, in one kernel: Both intra-node NVLink and inter-node RDMA live inside the same persistent kernel. Fine-grained intra-kernel overlap: Compute and communication overlap at tile/chunk granularity, covering both intra-node and inter-node GPU communication. Persistent kernel with SM specialization: CTAs self-assign roles: compute, intra-comm, inter-send, inter-reduce. The number of SMs dedicated to each role is tunable per shape. GPU-driven networking built on libibverbs: mKernel uses GPU-initiated RDMA writes without depending on NCCL or NVSHMEM. The communication backend is written from scratch to maximize performance and support heterogeneous networking devices. The Five Fused Kernels Kernel What it fuses Description AllGather + GEMM AllGather → GEMM Each rank holds a shard of A. While ranks gather peers’ shards over NVLink/RDMA, the local GEMM consumes tiles as soon as they arrive. GEMM + AllReduce GEMM → AllReduce Computes C = A @ B and reduces partial outputs across all ranks in one launch. Output tiles are pushed into the reduction tree the instant they’re produced. MoE Dispatch + GEMM All-to-All dispatch → grouped GEMM Routes MoE tokens to their expert ranks (intra-node NVLink + inter-node all-to-all) and runs the per-expert grouped GEMM in the same kernel. Tokens are processed as soon as they land — no staging buffer round-trip. Ring Attention Ring KV exchange → FlashAttention Sequence-parallel attention across ranks. Each step rotates a KV chunk around the ring while the local FlashAttention consumes the previously-received chunk. Compute and the ring send/recv run concurrently inside a single persistent kernel. GEMM + ReduceScatter GEMM → ReduceScatter Computes C = A @ B and reduce-scatters the output. Each output tile is reduced and forwarded to its owning rank as soon as it is produced. Evaluation Setup The research team evaluated mKernel on two 2-node × 8-H200 clusters that differ only in their inter-node fabric: Testbed Nodes × GPUs Intra-node Inter-node transport NIC AWS EFA 2 × 8 H200 NVLink AWS EFA / SRD 16 × 200 Gb/s EFA per node ConnectX-7 2 × 8 H200 NVLink InfiniBand 8 × 400 Gb/s NVIDIA ConnectX-7 per node mKernel was benchmarked against NCCL, Triton-distributed, Flux, Mercury, MagiAttention, Transformer-Engine, and ring-flash-attention. The team notes that further benchmarking at larger scale is still in progress. Backends and Requirements mKernel supports two networking backends: Backend Macro Transport Where it runs CX7 -DINTERNODE_BACKEND_IBVERBS libibverbs RC ConnectX-7 / InfiniBand / RoCE EFA -DINTERNODE_BACKEND_EFA libibverbs + efadv (SRD) AWS p5/p5e (H200, EFA) Both backends share the same host-side API and the same on-GPU kernel. Only the proxy/session implementation differs (session.h for CX7, session_efa.h for EFA). Requirements: NVIDIA Hopper GPUs (default build targets sm_90a), CUDA 12.9, Python with PyTorch. The CX7 backend requires libibverbs development headers and libraries. The EFA backend requires AWS EFA installation with libfabric, libibverbs, efadv, and EFA headers under EFA_HOME=/opt/amazon/efa by default. Marktechpost’s Visual Explainer UCCL mKernel — Multi-GPU, Multi-Node Fused Kernels Guide 01 / 07 — Overview What is mKernel? mKernel is an open-source library of persistent CUDA kernels from UC Berkeley’s UCCL project. It fuses intra-node NVLink communication, inter-node RDMA, and dense compute into a single kernel. Most existing fused kernel libraries operate within a single node or a single GPU. mKernel is designed from the start to span node boundaries. 43.6% of forward pass consumed by communication in production 47% of total execution time in popular MoE models 32% of end-to-end training time consumed by communication 02 / 07 — The Problem Why Host-Driven Communication Falls Short The standard model is host-driven: the CPU calls NCCL or NVSHMEM, which issues collective operations across GPUs. The UCCL team identifies two problems. CPUs are not scaling with GPUs. A GB300 NVL72 rack delivers 720 PFLOP/s FP8/FP6 and 1.44 EFLOP/s FP4. At those speeds, microsecond-scale overhead from cudaLaunchKernel, CPU-side sync checks, and inter-stream events shows up directly as pipeline bubbles. Overlap is too coarse. Host-driven systems overlap compute and communication only at kernel boundaries. Finer-grained overlap at the tile or chunk level is not possible from the host side. The answer: GPU-driven communication. The GPU itself triggers fine-grained transfers, fused into the same kernel as the compute. 03 / 07 — Design Four Core Design Properties 🖧 Multi-GPU + multi-node, in one kernel. Intra-node NVLink and inter-node RDMA both live

This year, there’s been a wave of notable energy companies going public via IPO in the US. The solar and battery company Solv Energy went public in February, to the tune of $6 billion. X-energy, which is building small modular nuclear reactors, did the same in April, and its stocks surged on its first day of trading to hit a $11.5 billion market cap. Most recently, the geothermal company Fervo Energy went public in mid-May, and its market cap is now about $12.4 billion. Those are all success stories in the IPO world. And it certainly doesn’t feel like a coincidence that all these companies are racing to provide electricity in an era of rising demand (partly due to data centers). Let’s take a look at how these firms are doing, what this moment says about the grid, and what’s coming next.  Let’s start with Fervo Energy, a company we’ve covered a lot over the years that’s working to develop enhanced geothermal energy. (We included it on our 2025 list of Climate Tech Companies to Watch.) While conventional geothermal requires finding specific spots with hot rock, water, and fractures to support a power plant, Fervo essentially uses fracking techniques to create the necessary conditions. The company was founded in 2017, and it raised about $1.5 billion from investors over the years before its IPO. Fervo’s first commercial project, Cape Station in Utah, is expected to have a capacity of about 500 megawatts. The first unit is set to start generating power for customers by October and the next two units by January 2027. The new funding from the IPO could help the company scale. Fervo currently has over 600 megawatts’ worth of binding power purchase agreements. And it has leases for land that could together generate more than 40 gigawatts of electricity. (As of 2024, the entire US geothermal fleet had a capacity of just 4 gigawatts.) The company also has an eye on cutting construction and drilling costs—its Cape Station plant is expected to cost about $7 per kilowatt, which is cheaper than new nuclear power plants but over twice the expense of building a new natural-gas plant in the US.  X-energy also aims to provide reliable clean power: it’s part of the wave of next-generation nuclear companies working on small modular reactors. The company is building high-temperature gas-cooled reactors, which flow helium over self-contained pebbles of nuclear fuel. These reactors will each generate 80 megawatts of electricity, less than one-tenth the output of larger ones like Unit 4 at Plant Vogtle in Georgia, the most recent addition to the commercial nuclear fleet in the US.   X-energy also saw its IPO go well, and prices surged in trading after the initial offering. One interesting tidbit here—the company had previously planned to go public in 2023 but decided against it because of difficult market conditions. The company is still years away from demonstrating its technology in a commercial project.  You may recall a story I wrote last year about its effort to build nuclear reactors at the site of a Dow Chemical plant in Texas. The company recently received a key environmental approval for that project, though it’s still waiting for the final green light from the Nuclear Regulatory Commission to start construction. Finally, Solv Energy builds solar and energy storage projects, mostly for utilities and independent power producers. Solar and batteries are some of the cheapest and easiest technologies to add to the grid, so this one could get a lot of capacity online, quickly. The company already has 21 gigawatts’ worth of projects operational across 35 states. Many companies in the energy sector are pinning their hopes on the rapid growth in data center construction and operation. The AI boom has transformed the energy landscape, pushing electricity demand higher in a country where it’s been relatively flat for the last decade or so. Solv Energy mentioned data centers over a dozen times in documents filed with the Securities and Exchange Commission before its IPO.  And Fervo and X-energy are particularly connected to the tech giants driving AI. Google has been a longtime investor in Fervo and also pioneered what it calls its clean transition tariff with the company. Amazon is a client of X-energy as well as an investor; it reportedly owns close to 20% of the company. Fervo and X-energy are also in industries that occupy a political sweet spot. President Trump and his administration have gone after wind power and other renewables, cutting off existing support and slowing approvals for new projects. Meanwhile, geothermal and particularly nuclear power have kept favor with the federal government and enjoyed continued tax credits and grant funding. If a few big leaders cash through these IPOs, it could help investors feel more confident about supporting the energy sector, even if that money is concentrated in later-stage ventures like these rather than earlier-stage companies.  We could see other firms, particularly in nuclear and geothermal, attempt a similar route in the year ahead. A key thing to watch here will be whether Fervo and X-energy in particular can succeed in scaling up and deploying their technology. If either of these companies stumbles or misses a timeline, it could have ripple effects for those hoping to follow in these very lucrative footsteps.  This article is from The Spark, MIT Technology Review’s weekly climate newsletter. To receive it in your inbox every Wednesday, sign up here. 

It is one thing to say AI will change the world. It is another to expect the class of 2026 to applaud it. In fact, when former Google CEO Eric Schmidt told University of Arizona graduates that their task is to help shape AI, he was met with a resounding chorus of boos. “I can hear you,” he said, before conceding that fears about disappearing jobs and a broken future were “rational.” This is not exactly the message one hopes to hear while sweating under a polyester gown and tallying student loan payments. Graduates have been jeering at AI pep talks at other commencements too, including ceremonies at the University of Central Florida and Middle Tennessee State University. Still, increasingly loud skepticism hasn’t stopped OpenAI from winning court cases, raising enormous sums of money, and launching new partnerships. And AI is even earning some unlikely cheerleaders: Reese Witherspoon has warned women to embrace it or be replaced by it.

Perplexity AI’s research team reimplemented their Unigram tokenizer from scratch in Rust and open-sourced the code in pplx-garden, their inference technology repository. At production input lengths, the new encoder cuts p50 latency by roughly 5x versus the Hugging Face tokenizers crate, ~2x versus SentencePiece (C++), and ~1.5x versus IREE’s tokenizer (C), with zero steady-state heap allocations. In production, it reduced CPU utilization in Perplexity’s inference stack by 5-6x and shaved double-digit milliseconds off reranker latency. Why Tokenization Became a Bottleneck LLM inference cost is typically framed around GPU work: KV caches, attention kernels, expert routing. But smaller models, such as embedding models, classifiers, and rerankers, tell a different story. These models are two to three orders of magnitude smaller than frontier transformers. A reranker scoring hundreds of candidate documents per request is a clear example. With a small model, GPU compute often finishes in single-digit milliseconds. Every input still passes through CPU-side tokenization first. When batch sizes are large, tokenization becomes a meaningful fraction of total request latency. Perplexity’s work targets XLM-RoBERTa, a model with a 250K-token Unigram vocabulary trained with SentencePiece. Fine-tuned RoBERTa-family encoders are a common production choice for ranking, retrieval, and similarity tasks. What is Unigram Tokenization? Unigram tokenization was introduced by Kudo in 2018 and is implemented in SentencePiece. It frames segmentation as a most-probable-path problem. Each vocabulary token has a learned log-probability. The tokenizer picks the segmentation whose token scores sum to the highest value. The algorithm used to find that best path is the Viterbi algorithm, a dynamic programming technique from 1967. Byte positions form graph layers and vocabulary tokens are edges spanning a contiguous byte range. The DP recurrence iterates over byte positions and updates the best-scoring path at each position. The outer loop runs in linear time relative to input length. The inner loop walks a vocabulary trie (a prefix tree structure) at each byte position. On a 16K-token input, this inner walk executes hundreds of thousands of trie transitions. It is the hot path. What was Slow in the Hugging Face Implementation The Hugging Face tokenizers crate is the default Rust tokenizer most teams reach for. Perplexity used it as the benchmark reference. At 514 tokens (512 + BOS/EOS injection), the reference implementation had three costly patterns: Bottleneck Mechanism Measured impact Allocation per match String::from_utf8 + AHashMap lookup per trie match 7,295 allocations at 514 tokens; 299,171 at 16K Pointer chase per byte AHashMap at every trie node; 4 dependent loads per byte step Dependent-load latency dominates the hot path L2 thrashing on long inputs DP table and output buffers freshly allocated each call L2 miss rate climbs from 8% at 128 tokens to 50% at 16K Per-token allocation is constant: roughly 2 KB and ~18 allocations per token, regardless of input size. The latency problem becomes severe at longer inputs when cumulative allocations overflow the per-core L2 cache. Establishing a Baseline Before Changing the Trie Before switching the trie structure, Perplexity first isolated how much cost came from unnecessary work alone. They made a zero-allocation port of the reference: same HashMap trie, but with a caller-owned scratch struct reused across calls and token IDs stored directly in trie nodes (removing the per-match string allocation and secondary hash-map lookup). This baseline already cut p50 latency to 155 µs at 514 tokens, down from 326 µs in the reference. Instructions retired dropped 2.4x. The remaining cost was the HashMap pointer chase itself, which the next step addressed. The Three Optimizations Optimization 1: Double-Array Trie The Hugging Face trie stores children in a HashMap at every node. Each byte step requires a hash computation, two pointer dereferences, and a heap access. Perplexity replaced this with a double-array trie, the same structure used by SentencePiece and IREE, originally introduced by Aoe in 1989. A double-array trie encodes the entire trie in two flat integer arrays, base and check. A child lookup is: next = base[node] + byte, then verify check[next] == node. That is two array reads, one integer add, and one comparison, with no hashing and no pointer chasing. For XLM-RoBERTa’s 250K vocab, the whole trie fits in ~9 MB of contiguous memory. The hot working set per encode is on the order of 100 KB, which fits in L2 cache. Unlike SentencePiece and IREE, which are general-purpose libraries with lattice bookkeeping and multi-stage pipelines, Perplexity inlined the trie directly in the Viterbi loop and dropped that overhead entirely. Result at 514 tokens: p50 dropped from 155 µs (zero-allocation baseline) to 68 µs. Wall-clock fell 4.8x from the original reference. Optimization 2: Bitmap and Inline Packing The double-array trie still requires two dependent array loads per byte step: first the parent’s base offset, then the check array to confirm the transition is valid. Perplexity replaced the check array with a per-node bitmap (four 64-bit words, 32 bytes) that records which of the 256 possible bytes have valid child transitions. A bitmap lookup compiles to a single bit test against one 64-bit word. The check array is used only during trie construction and dropped from the runtime layout entirely. They also packed all four per-node fields (bitmap, base, token ID, and score) into a single 64-byte cache line, matching CPU cache line width exactly. One trie step now loads a single cache line covering the bitmap for the next-byte check, the base offset for the child slot, and the token ID and score at terminal nodes. Trade-off: trie size grows from ~9 MB to ~50 MB (780K nodes x 64 bytes). The hot working set per encode remains ~100 KB. Result at 514 tokens: Additional 4.5% wall-clock reduction. L2 accesses dropped from 4.6K to 1.8K per encode. Optimization 3: Huge Pages for the Trie At 50 MB, the trie spans roughly 12,000 virtual pages on a default Linux system using 4 KB pages. The first-level data TLB on Intel Sapphire Rapids holds 96 entries. Each Viterbi step touches a different trie node, so TLB misses accumulate. Over a 512-token encode, Perplexity estimated roughly

arXiv:2605.26849v1 Announce Type: new Abstract: Sampling multiple responses improves language model reasoning, but uniform compute allocation is inefficient: easy questions are over-sampled while hard questions remain under-explored. We propose Uncertainty-Aware Budget Allocation (UAB), a concave integer optimization framework that reallocates a fixed sampling budget based on per-question uncertainty estimated at no additional inference cost. In Phase 1, every question receives one generation; its average negative log-likelihood (ANLL), extracted directly from output log-probabilities, serves as a difficulty signal while the generation contributes to the final vote. In Phase 2, the remaining budget is allocated by a marginal-greedy algorithm that solves a concave coverage-maximization surrogate exactly: uncertain questions receive more sampling budget while confident questions receive fewer additional samples. Evaluated on six open-weight and black-box models spanning 1.5B to 27B parameters and five reasoning benchmarks covering math, logic, and preference tasks, UAB outperforms baselines by up to +3% in average accuracy and up to +5% on individual benchmarks, with the largest gains in low-resource settings, requiring no auxiliary model or additional LLM call. Code is publicly available at https://github.com/manhitv/UAB.