Distributed training infrastructure

Two different reasons to go distributed

“Distributed training” gets used as one phrase for two genuinely different problems. The first is a speed problem: your dataset is large enough that training on one GPU would take weeks, and you want to cut that down by spreading the work across many GPUs running in parallel. The second is a capacity problem: your model’s parameters, activations, and optimizer state no longer fit in one GPU’s memory at all, regardless of how much time you’re willing to spend — training simply cannot start on a single device. These call for different solutions, and a lot of confusion in this space comes from treating “distributed training” as one technique instead of recognizing which problem you actually have.

Data parallelism: same model, different data

Data parallelism is the answer to the speed problem. The full model is replicated identically on every GPU; each replica trains on a different shard of the data in each step, computing its own forward pass and its own gradients. Those gradients are then synchronized across all GPUs — usually averaged — so every replica applies the same update and stays identical before the next step begins. This is conceptually simple and scales well when the model fits comfortably on one device: doubling the number of GPUs roughly doubles the data throughput, in the ideal case. The catch is that synchronization step. Averaging gradients across, say, 64 GPUs requires real communication, and if that communication is slow relative to how fast each GPU finishes its local computation, the GPUs sit idle waiting for each other instead of training. This is why data-parallel training lives or dies on the quality of the interconnect between GPUs, not just on how many GPUs you add.

Model parallelism: splitting the model itself

Model parallelism is the answer to the capacity problem. When a model is too large to replicate on a single GPU — common with large language models whose parameters alone can exceed one device’s memory — the model is split across multiple GPUs instead, with each one holding a different subset of layers (pipeline parallelism) or a different slice of each layer’s weights (tensor parallelism). A forward pass now has to physically cross GPU boundaries: GPU 1 computes its layers and sends activations to GPU 2 to continue, and gradients flow back the same path during backpropagation. This solves the memory problem but introduces its own bottleneck — GPUs further along the pipeline sit idle waiting for earlier ones to finish, unless the work is carefully pipelined so multiple batches are in flight at once. In practice, large-scale training often combines both: model parallelism to fit an oversized model across a group of GPUs, and data parallelism across multiple such groups to get throughput on top of that.

The infrastructure that makes this actually work

Neither form of parallelism is free, and both depend on infrastructure most single-GPU training never has to think about. High-bandwidth, low-latency interconnects between GPUs — technologies like NVLink within a machine and InfiniBand or specialized networking fabrics between machines — exist because ordinary Ethernet is far too slow for the volume of gradient and activation traffic distributed training generates. An orchestration layer is needed to place workers correctly (which GPU gets which model shard or data shard), to restart a failed worker without corrupting the whole training job’s state, and to checkpoint progress so a multi-day run survives a single node failure instead of losing everything. None of this is optional extra polish — without it, distributed training is either too slow to be worth the extra hardware or too fragile to complete a long training run at all.

Why this matters going forward

Distributed training is where the infrastructure decisions earlier in this group stop being abstract. Whether you provisioned your own cluster with infrastructure as code or you’re using a managed platform’s distributed training support, the actual performance you get depends on understanding which bottleneck you’re solving — data volume or model size — and whether the interconnect and orchestration underneath can keep GPUs fed with work instead of waiting on each other. Getting this wrong doesn’t just waste money on idle GPUs; past a certain scale, it’s the difference between a training run that finishes and one that never converges at all.