{"id":53092,"date":"2025-11-22T08:13:57","date_gmt":"2025-11-22T08:13:57","guid":{"rendered":"https:\/\/youzum.net\/perplexity-ai-releases-transferengine-and-pplx-garden-to-run-trillion-parameter-llms-on-existing-gpu-clusters\/"},"modified":"2025-11-22T08:13:57","modified_gmt":"2025-11-22T08:13:57","slug":"perplexity-ai-releases-transferengine-and-pplx-garden-to-run-trillion-parameter-llms-on-existing-gpu-clusters","status":"publish","type":"post","link":"https:\/\/youzum.net\/zh\/perplexity-ai-releases-transferengine-and-pplx-garden-to-run-trillion-parameter-llms-on-existing-gpu-clusters\/","title":{"rendered":"Perplexity AI Releases TransferEngine and pplx garden to Run Trillion Parameter LLMs on Existing GPU Clusters"},"content":{"rendered":"

How can teams run trillion parameter language models on existing mixed GPU clusters without costly new hardware or deep vendor lock in? Perplexity\u2019s research team has released TransferEngine and the surrounding pplx garden toolkit as open source infrastructure for large language model systems. This provides a way to run models with up to 1 trillion parameters across mixed GPU clusters, without locking into a single cloud provider or buying new GB200 class hardware.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.27656<\/figcaption><\/figure>\n<\/div>\n

The real bottleneck, network fabrics not FLOPs<\/strong><\/h3>\n

Modern deployments of Mixture of Experts models such as DeepSeek V3 with 671 billion parameters and Kimi K2 with 1 trillion parameters no longer fit on a single 8 GPU server<\/a>. They must span multiple nodes, so the main constraint becomes the network fabric between GPUs.<\/p>\n

Here the hardware landscape is fragmented. NVIDIA ConnectX 7 typically uses Reliable Connection transport with in order delivery. AWS Elastic Fabric Adapter uses Scalable Reliable Datagram transport that is reliable but out of order, and a single GPU may need 4 network adapters at 100 Gbps, or 2 at 200 Gbps, to reach 400 Gbps.<\/p>\n

Existing libraries such as DeepEP, NVSHMEM, MoonCake and NIXL tend to optimize for one vendor and degrade or lack support on the other side. Perplexity\u2019s research team directly states in the research paper<\/a> that there was no viable cross provider solution for LLM inference before this work.<\/p>\n

TransferEngine, a portable RDMA layer for LLM systems<\/strong><\/h3>\n

TransferEngine addresses this by targeting only the intersection of guarantees across Network Interface Controllers. It assumes that the underlying RDMA transport is reliable, but does not assume any ordering of messages. On top of this, it exposes one sided WriteImm operations and an ImmCounter primitive for completion notification.<\/p>\n

The library<\/a> provides a minimal API in Rust. It offers two sided Send and Recv for control messages, and three main one sided operations, submit_single_write<\/code>, submit_paged_writes<\/code>, and submit_scatter<\/code>, plus a submit_barrier<\/code> primitive for synchronization across a group of peers. A NetAddr structure identifies peers and an MrDesc structure describes registered memory regions. An alloc_uvm_watcher call creates a device side watcher for CPU GPU synchronization in advanced pipelines.<\/p>\n

Internally, TransferEngine spawns one worker thread per GPU and builds a DomainGroup per GPU that coordinates between 1 and 4 RDMA Network Interface Controllers. A single ConnectX 7 provides 400 Gbps. On EFA, the DomainGroup aggregates 4 network adapters at 100 Gbps, or 2 at 200 Gbps, to reach the same bandwidth. The sharding logic knows about all Network Interface Controllers and can split a transfer across them.<\/p>\n

Across hardware, the research team reports peak throughput of 400 Gbps on both NVIDIA ConnectX 7 and AWS EFA. This matches single platform solutions and confirms that the abstraction layer does not leave large performance on the table.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.27656<\/figcaption><\/figure>\n<\/div>\n

pplx garden, the open source package<\/strong><\/h3>\n

TransferEngine ships as part of the pplx garden repository on GitHub under an MIT license. The directory structure is straightforward. fabric-lib<\/code> contains the RDMA TransferEngine library, p2p-all-to-all<\/code> implements a Mixture of Experts all to all kernel, python-ext<\/code> provides the Python extension module from the Rust core, and python\/pplx_garden<\/code> contains the Python package code.<\/p>\n

The system requirements reflect a modern GPU cluster. Perplexity research team recommends Linux kernel 5.12 or newer for DMA BUF support, CUDA 12.8 or newer, libfabric, libibverbs, GDRCopy, and an RDMA fabric with GPUDirect RDMA enabled. Each GPU should have at least one dedicated RDMA Network Interface Controller.<\/p>\n

Disaggregated prefill and decode<\/strong><\/h3>\n

The first<\/strong> production use case is disaggregated inference. Prefill and decode run on separate clusters, so the system must stream KvCache from prefill GPUs to decode GPUs at high speed.<\/p>\n

TransferEngine uses alloc_uvm_watcher to track progress in the model. During prefill, the model increments a watcher value after each layer\u2019s attention output projection. When the worker observes a change, it issues paged writes for the KvCache pages of that layer, followed by a single write for the remaining context. This approach allows layer by layer streaming of cache pages without fixed world membership, and it avoids the strict ordering constraints of collectives.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.27656<\/figcaption><\/figure>\n<\/div>\n

Fast weight transfer for reinforcement learning<\/strong><\/h3>\n

The second<\/strong> system is asynchronous reinforcement learning fine tuning, where training and inference run on separate GPU pools. Traditional designs gather updated parameters to a single rank then broadcast them, which limits throughput to one Network Interface Controller.<\/p>\n

Perplexity research team instead uses TransferEngine to perform point to point weight transfer. Each training GPU writes its parameter shard directly into the corresponding inference GPUs using one sided writes. A pipelined execution splits each tensor into stages, host to device copy when Fully Sharded Data Parallel offloads weights, reconstruction and optional quantization, RDMA transfer, and a barrier implemented through scatter and ImmCounter.<\/p>\n

In production, this setup delivers weight updates for models such as Kimi K2 at 1 trillion parameters and DeepSeek V3 at 671 billion parameters in about 1.3 seconds from 256 training GPUs to 128 inference GPUs.<\/p>\n

\n
\"\"
https:\/\/arxiv.org\/pdf\/2510.27656<\/figcaption><\/figure>\n<\/div>\n

Mixture of Experts routing across ConnectX and EFA<\/strong><\/h3>\n

The third<\/strong> piece in pplx garden is a point to point Mixture of Experts dispatch and combine kernel. It uses NVLink for intra node traffic and RDMA for inter node traffic. Dispatch and combine are split into separate send and receive phases so that the decoder can micro batch and overlap communication with grouped general matrix multiply.<\/p>\n

A host proxy<\/a> thread polls GPU state and calls TransferEngine when send buffers are ready. Routes are exchanged first, then each rank computes contiguous receive offsets for each expert and writes tokens into private buffers that can be reused between dispatch and combine. This reduces memory footprint and keeps writes large enough to use the full link bandwidth.<\/p>\n

On ConnectX 7, Perplexity research team reports state of the art decode latency that is competitive with DeepEP across expert counts. On AWS EFA, the same kernel delivers the first viable MoE decode latencies with higher but still practical values. <\/p>\n

In multi node tests with DeepSeek V3 and Kimi K2 on AWS H200 instances, distributing the model across nodes reduces latency at medium batch sizes, which is the common regime for production serving.<\/p>\n

Comparison Table<\/strong><\/h3>\n
\n\n\n\n\n\n\n\n\n
Key point<\/th>\nTransferEngine (pplx garden)<\/th>\nDeepEP<\/th>\nNVSHMEM (generic MoE use)<\/th>\nMooncake<\/th>\n<\/tr>\n<\/thead>\n
Primary role<\/td>\nPortable RDMA point to point for LLM systems<\/td>\nMoE all to all dispatch and combine<\/td>\nGeneral GPU shared memory and collectives<\/td>\nDistributed KV cache for LLM inference<\/td>\n<\/tr>\n
Hardware focus<\/td>\nNVIDIA ConnectX 7 and AWS EFA, multi NIC per GPU<\/td>\nNVIDIA ConnectX with GPU initiated RDMA IBGDA<\/td>\nNVIDIA GPUs on RDMA fabrics including EFA<\/td>\nRDMA NICs in KV centric serving stacks<\/td>\n<\/tr>\n
EFA status<\/td>\nFull support, peak 400 Gbps reported<\/td>\nNo support, requires IBGDA on ConnectX<\/td>\nAPI works but MoE use shows severe degradation on EFA<\/td>\nPaper reports no EFA support in its RDMA engine<\/td>\n<\/tr>\n
Portability for LLM systems<\/td>\nCross vendor, single API across ConnectX 7 and EFA<\/td>\nVendor specific and ConnectX focused<\/td>\nNVIDIA centric, not viable for EFA MoE routing<\/td>\nFocused on KV sharing, no cross provider support<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

Key Takeaways<\/strong><\/h3>\n