Inference speed is becoming a competitive metric for large language models. Xiaomi\u2019s MiMo team just released MiMo-V2.5-Pro-UltraSpeed, built in collaboration with the TileRT systems group. It decodes faster than 1000 tokens per second on a 1-trillion-parameter model. Xiaomi team describes this as a first at trillion-parameter scale. Demos show generation peaks near 1200 tokens per second. The notable part is the hardware: it runs on commodity GPUs, not custom silicon.<\/p>\n
UltraSpeed is a high-speed serving mode for the existing MiMo-V2.5-Pro model. The base model uses a Mixture-of-Experts (MoE) architecture at trillion-parameter scale. UltraSpeed targets generation speed rather than model capability. It changes how fast the model produces output tokens. The speedup comes from three coordinated techniques across the model and the serving system. Xiaomi calls this approach extreme model-system codesign. Crucially, the entire stack runs on a single standard 8-GPU commodity node.<\/p>\n
The first layer is FP4 quantization. At trillion scale, FP8 or FP16 weights create heavy memory and bandwidth pressure. Lower bit-width weights move through memory faster, which directly lifts decode speed. Xiaomi uses the MXFP4 format, applied selectively to the MoE Experts only. Other modules keep higher precision, reported as FP8 by TileRT. Experts hold most parameters and tolerate quantization best, so the tradeoff is favorable. Quantization-Aware Training (QAT) keeps benchmark quality essentially on par with the original.<\/p>\n
The second layer is DFlash speculative decoding, covered in detail below. The third layer is TileRT, the system that executes everything on the GPU. Each technique alone is not enough. The 1000 TPS result needs all three aligned tightly.<\/p>\n
Standard speculative decoding uses a small draft model to guess upcoming tokens. The large model then verifies those guesses in parallel. Rejection sampling keeps output identical to normal decoding, so quality is lossless. The problem is that the draft model still generates tokens one at a time. DFlash, a method from the research community, removes that constraint. It uses block-level masked parallel prediction. The draft model fills a whole block of masked positions in one forward pass.<\/p>\n
Xiaomi tuned DFlash with the Muon second-order optimizer and model self-distillation. The draft model uses Sliding Window Attention (SWA) only, matching the MiMo-V2 design. This makes per-prediction compute constant rather than growing with context length. Block size is capped at 8 to limit verification cost and raise concurrency.<\/p>\n
Acceptance length measures how many draft tokens survive verification each round.<\/p>\n In coding, six to seven of eight draft tokens are accepted per round. Some samples reach a maximum of 7.14. <\/p>\n At 1000 TPS, each operator runs for only microseconds. Traditional systems launch operators one by one, and each launch costs time. Those gaps fracture the execution stream and become the real bottleneck. TileRT replaces this with a Persistent Engine Kernel that stays resident on the GPU. It uses Warp Specialization to split data movement, compute, and communication into coordinated roles. Small operations like RMSNorm, RoPE, and KV cache writes turn into bottlenecks at this scale. The system was co-designed with the FP4 and DFlash choices, not added afterward.<\/p>\n The release targets latency-sensitive work where waiting breaks the loop:<\/p>\n These are throughput-bound workloads where raw token speed is the binding constraint.<\/p>\n The first table contrasts the two routes to extreme decode speed.<\/strong><\/p>\n The second table compares the standard model with the UltraSpeed mode.<\/strong><\/p>\n\n\n
\n \nScenario<\/th>\n Acceptance Length<\/th>\n<\/tr>\n<\/thead>\n \n Coding<\/td>\n 6.30<\/td>\n<\/tr>\n \n Math \/ Reasoning<\/td>\n 5.56<\/td>\n<\/tr>\n \n Agent<\/td>\n 4.29<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n TileRT: Squeezing the Microseconds<\/strong><\/h2>\n
Use Cases<\/strong><\/h2>\n
\n
How It Compares<\/strong><\/h2>\n
\n\n
\n \nApproach<\/th>\n Hardware<\/th>\n How speed is achieved<\/th>\n<\/tr>\n<\/thead>\n \n Cerebras<\/td>\n Wafer-Scale integration (custom)<\/td>\n Scale on a single custom wafer<\/td>\n<\/tr>\n \n Groq<\/td>\n Custom architecture<\/td>\n Pure on-chip SRAM<\/td>\n<\/tr>\n \n MiMo \u00d7 TileRT<\/td>\n Commodity GPUs (8-GPU node)<\/td>\n Model-system codesign: FP4 + DFlash + TileRT<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n