{"id":85785,"date":"2026-04-24T15:27:25","date_gmt":"2026-04-24T15:27:25","guid":{"rendered":"https:\/\/youzum.net\/google-deepmind-introduces-decoupled-diloco-an-asynchronous-training-architecture-achieving-88-goodput-under-high-hardware-failure-rates\/"},"modified":"2026-04-24T15:27:25","modified_gmt":"2026-04-24T15:27:25","slug":"google-deepmind-introduces-decoupled-diloco-an-asynchronous-training-architecture-achieving-88-goodput-under-high-hardware-failure-rates","status":"publish","type":"post","link":"https:\/\/youzum.net\/zh\/google-deepmind-introduces-decoupled-diloco-an-asynchronous-training-architecture-achieving-88-goodput-under-high-hardware-failure-rates\/","title":{"rendered":"Google DeepMind Introduces Decoupled DiLoCo: An Asynchronous Training Architecture Achieving 88% Goodput Under High Hardware Failure Rates"},"content":{"rendered":"

Training frontier AI models is, at its core, a coordination problem. Thousands of chips must communicate with each other continuously, synchronizing every gradient update across the network. When one chip fails or even slows down, the entire training run can stall. As models scale toward hundreds of billions of parameters, that fragility becomes increasingly untenable. Google DeepMind is now proposing a different model entirely.<\/p>\n

Google DeepMind researchers introduced Decoupled DiLoCo<\/strong> (Distributed Low-Communication), a distributed training architecture that decouples compute into asynchronous, fault-isolated \u2018islands,\u2019 enabling large language model pre-training across geographically distant data centers without requiring the tight synchronization that makes conventional approaches brittle at scale.<\/p>\n

The Problem with Traditional Distributed Training<\/strong><\/h3>\n

To understand why Decoupled DiLoCo is important, it helps to understand how distributed training typically works. Standard Data-Parallel<\/strong> training replicates a model across many accelerators (GPUs or TPUs), each processing a different mini-batch of data. After each forward and backward pass, gradients must be averaged across every device \u2014 a process called AllReduce<\/strong> \u2014 before the next training step can begin. This blocking synchronization step means every device must wait for the slowest one. Across thousands of chips spanning multiple data centers, that bottleneck is not just inconvenient; it makes global-scale training effectively impractical.<\/p>\n

Bandwidth is another hard constraint. Conventional Data-Parallel training requires approximately 198 Gbps of inter-datacenter bandwidth across eight data centers \u2014 far beyond what standard wide-area networking (WAN) can support between geographically distributed facilities.<\/p>\n

How Decoupled DiLoCo Works<\/strong><\/h3>\n

Decoupled DiLoCo builds on two prior systems from Google. The first is Pathways<\/strong>, which introduced a distributed AI system based on asynchronous data flow, allowing different compute resources to work at their own pace without blocking on one another. The second is DiLoCo<\/strong>, which dramatically reduced the inter-datacenter bandwidth required for distributed training by having each worker perform many local gradient steps before communicating with peers \u2014 dramatically reducing how much data needs to flow between data centers.<\/p>\n

Decoupled DiLoCo brings both ideas together. Built on top of Pathways, training is divided across separate clusters of accelerators called learner units<\/strong> \u2014 the \u2018islands\u2019 of compute. Each learner unit trains semi-independently, performing many local steps, before sharing a compressed gradient signal with an outer optimizer that aggregates updates across all learner units. Because this outer synchronization step is asynchronous, a chip failure or slow learner unit in one island does not block the others from continuing to train.<\/p>\n

The bandwidth savings are dramatic. Decoupled DiLoCo reduces required inter-datacenter bandwidth from 198 Gbps to just 0.84 Gbps across eight data centers \u2014 multiple orders of magnitude lower \u2014 making it compatible with standard internet-scale connectivity between datacenter facilities rather than requiring custom high-speed network infrastructure.<\/p>\n

Self-Healing Through Chaos Engineering<\/strong><\/h3>\n

One of the most technically significant properties of Decoupled DiLoCo is its fault tolerance. The research team used chaos engineering<\/strong>, a method that deliberately introduces artificial hardware failures into a running system to test its robustness during training runs. The system continued training after the loss of entire learner units, and then seamlessly reintegrated those units when they came back online. This behavior is what the research team describes as \u2018self-healing\u2019.<\/p>\n

In simulations involving 1.2 million chips under high failure rates, Decoupled DiLoCo maintained a goodput<\/strong> (the fraction of time the system is performing useful training) of 88%, compared to just 27% for standard Data-Parallel methods. Goodput is the practical metric that matters here: a training run with high nominal compute but low goodput wastes significant resources.<\/p>\n

\n
\"\"
https:\/\/deepmind.google\/blog\/decoupled-diloco\/?<\/figcaption><\/figure>\n<\/div>\n

Critically, these resilience gains come with minimal degradation in model quality. In real-world experiments using Gemma 4<\/strong> models, Decoupled DiLoCo achieved an average ML benchmark accuracy of 64.1%, compared to 64.4% for the conventional baseline \u2014 a difference well within the noise of typical evaluation variance.<\/p>\n

Training a 12B Model Across Four U.S. Regions<\/strong><\/h3>\n

The research team validated Decoupled DiLoCo at production scale by successfully training a 12 billion parameter model<\/strong> across four separate U.S. regions using just 2\u20135 Gbps of wide-area networking, a bandwidth level achievable with existing commercial internet infrastructure between data center facilities. The system accomplished this more than 20 times faster than conventional synchronization methods. The key reason: rather than forcing compute to pause and wait for communication to complete, Decoupled DiLoCo incorporates required communication into longer periods of computation, eliminating the \u201cblocking\u201d bottlenecks that make conventional distributed training slow at global scale.<\/p>\n

Mixing Hardware Generations<\/strong><\/h3>\n

An underappreciated implication of the architecture is its support for heterogeneous hardware. Because learner units operate asynchronously, they do not need to run on identical hardware at the same clock speed. The research team demonstrated training runs that mixed TPU v6e<\/strong> and TPU v5p<\/strong> chips \u2014 different hardware generations with different performance characteristics \u2014 in a single training job, without degrading ML performance relative to homogeneous runs.<\/p>\n

This has two practical consequences worth noting. First, it extends the useful life of existing hardware, allowing older accelerators to continue contributing meaningfully to large-scale training. Second, because new hardware generations do not arrive everywhere at once, being able to train across generations can alleviate the recurring logistical and capacity bottlenecks that arise during hardware transition periods \u2014 a real operational challenge at organizations running large training infrastructure.<\/p>\n

Key Takeaways<\/strong><\/h3>\n