Mixture of experts

Decoupling size from cost

Every model covered so far in this roadmap is dense: every parameter participates in processing every token. A mixture-of-experts (MoE) model breaks this coupling deliberately. Instead of one set of parameters that every token passes through, the model is organised into many smaller sub-networks called experts, and a learned router decides, for each individual token, which small subset of experts should actually process it — commonly just one or two out of dozens or more available.

Why this changes the serving tradeoff

In a dense model, doubling the parameter count roughly doubles the compute required per token, because every added parameter sits on the path every token travels. In an MoE model, adding more experts increases the model’s total parameter count without increasing the compute cost per token at all, as long as the number of experts activated per token stays fixed. This is the entire appeal: a much larger total model, with much more capacity spread across its experts, running at close to the compute cost of a far smaller dense model.

What to watch for

This AI Engineer talk covers the serving-specific angle this lesson needs directly: how the router actually picks which experts handle a given token, and the total-versus-active-parameter distinction that drives the real compute and memory tradeoff at serving time — the framing that matters for running these models, rather than only the architectural idea in the abstract.

The memory bill MoE does not avoid

The saving MoE buys is in compute, not memory. Every expert has to sit resident in GPU memory at all times, because the router’s decision happens per token, at runtime, and there is no way to know in advance which experts any given request will need — predicting that and loading experts on demand would introduce its own serving complexity and latency cost. This means an MoE model’s memory footprint scales with its full parameter count, including every expert whether or not it happens to be used by the token currently being processed, even though its compute cost scales only with however many experts are actually activated.

Why the router itself is worth paying attention to

The router is a small learned component, separate from the experts themselves, that decides which experts should handle a given token. Its quality directly determines whether an MoE model’s capacity is actually used well: a router that routes tokens evenly and sensibly across experts lets the model’s full capacity contribute meaningfully, while a poorly trained router that collapses onto favouring only a handful of experts effectively wastes most of the model’s extra parameters, since the unused experts contribute nothing to the tokens that never reach them. This routing decision also has a direct serving-time consequence: an unevenly loaded set of experts means some GPUs handling popular experts in an expert-parallel deployment sit under heavier load than others handling rarely-selected ones, which is its own load-balancing problem layered on top of everything else this roadmap covers about distributing work evenly across hardware.

What this means for serving hardware and scale

This memory-heavy, compute-light profile is exactly why MoE models are a natural fit for the model parallelism techniques covered later in this roadmap, and specifically for expert parallelism — spreading different experts across different GPUs, so that no single device has to hold every expert in memory at once, while routing each token’s computation to whichever device actually holds the expert it needs. An MoE model that would be far too large in total parameters to fit on one GPU can become practical to serve once its experts are distributed this way, which is a serving-architecture decision these models force in a way dense models never do.