{"id":109051,"date":"2026-08-04T20:07:39","date_gmt":"2026-08-04T20:07:39","guid":{"rendered":"https:\/\/youzum.net\/cursor-open-sources-mixture-of-kittens-mok-a-deterministic-moe-training-megakernel-for-gb300-nvl72-racks\/"},"modified":"2026-08-04T20:07:39","modified_gmt":"2026-08-04T20:07:39","slug":"cursor-open-sources-mixture-of-kittens-mok-a-deterministic-moe-training-megakernel-for-gb300-nvl72-racks","status":"publish","type":"post","link":"https:\/\/youzum.net\/ja\/cursor-open-sources-mixture-of-kittens-mok-a-deterministic-moe-training-megakernel-for-gb300-nvl72-racks\/","title":{"rendered":"Cursor Open-Sources Mixture-of-Kittens (MoK): A Deterministic MoE Training Megakernel for GB300 NVL72 Racks"},"content":{"rendered":"

Cursor Research has open-sourced Mixture-of-Kittens (MoK)<\/a>, the mixture-of-experts training megakernel behind its Composer<\/a> models. MoK fuses every MoE communication and computation step into a single deterministic kernel. Cursor team reports up to 2.37x higher throughput than the strongest public baseline. It already powers Composer training across tens of thousands of GPUs.<\/p>\n

Is it deployable<\/strong><\/h2>\n

Yes, but the hardware floor is high. MoK is on GitHub<\/a> under Apache-2.0. It requires NVIDIA Blackwell SM100 or SM103 GPUs, which means GB200 NVL72 or GB300 NVL72 racks. It also needs Python 3.12+, PyTorch 2.10+, and CUDA toolkit 13.0+. Inter-GPU buffers rely on PyTorch symmetric memory.<\/p>\n

That limits realistic adopters to organizations that own or rent NVL72 capacity. Frontier labs, funded model startups, GPU neoclouds, and national computing centers fit. Single-node teams and 8-GPU shops do not.<\/p>\n

Applications are narrow but high-value. They include pretraining and post-training of DeepSeek-V3-style MoE models. Determinism also makes it useful for on-policy RL post-training and internal ablations. Relevant industries are AI model development, cloud GPU infrastructure, code-generation tooling, and quantitative research.<\/p>\n

MoE layer as the bottleneck<\/strong><\/h2>\n

Cursor\u2019s earlier work covered the compute side. The research team wrote its own MXFP8 and NVFP4 training kernels<\/a> and a \u2018warp decode\u2019<\/a> path for MoE inference. Those assumed inter-GPU communication was handled separately.<\/p>\n

In production, communication became the limiting factor. The MoE layer can consume more than half of end-to-end training time. Moving to GB300 NVL72s changed the problem again. A rack is 72 GPUs inside one NVLink domain, which allows fine-grained overlap. But the integrated Grace CPUs are slow relative to the GPUs. CPU-GPU synchronization therefore has to be minimized aggressively.<\/p>\n

Three design decisions that matter<\/strong><\/h2>\n