Autoscaling and cold starts

Capacity that has to actually be ready, not just requested

Scaling a web service typically means starting a new, lightweight process that is ready to handle requests within moments. Scaling GPU inference capacity is a fundamentally slower operation, because a new instance is not actually useful until it has loaded the model’s entire set of weights — per model memory footprint, potentially hundreds of gigabytes — from storage into GPU memory. This loading time, the cold start, is the central problem this lesson covers, and it changes what “autoscaling” has to mean for inference compared to ordinary web infrastructure.

Why a slow cold start defeats naive autoscaling

The simplest autoscaling strategy — watch demand, and spin up a new instance the moment it rises — assumes new capacity becomes available quickly enough to matter. For a large model with a multi-minute cold start, this assumption breaks down: by the time a newly requested instance finishes loading and becomes ready to serve, the traffic spike that triggered it may already have overwhelmed existing capacity for several minutes, with users experiencing degraded service the whole time the new instance was still loading. The new capacity arrives, correctly provisioned, but too late to have actually helped with the problem it was meant to solve.

What to watch for

A focused conference talk about exactly why GPU cold starts are expensive — loading large model weights into VRAM takes real, substantial time — and how streaming weight loading specifically cuts that time down, which is the core technique this lesson needs explained concretely.

Techniques that shrink the gap

Several approaches exist to close the gap between demand arriving and capacity being ready. Streaming weight loading begins making an instance partially usable before every single weight has finished loading, rather than waiting for the complete load to finish before serving anything at all. Keeping a pool of warm standby instances — already loaded and ready, sitting idle until needed — trades a real ongoing cost (paying for GPU capacity that may not be used) for the ability to absorb a spike instantly, without waiting on any cold start at all. Predictive scaling, where traffic patterns are known or forecastable in advance, can start loading new capacity before demand actually arrives, converting what would have been a reactive cold start into one that finishes before it is needed. The deeper links cover two more approaches to this same problem in real depth: a Kubernetes-specific streaming approach, and Modal’s own engineering for making a cold start on their serverless GPU platform dramatically faster than the naive version.

Why this connects directly to the economics covered elsewhere

A cold start is not only a latency problem — it is directly a cost problem too, since keeping enough warm standby capacity to avoid cold starts entirely means paying for GPU time that sits idle between traffic spikes. Deciding how much standby capacity is worth maintaining, against how often and how severely cold starts would otherwise hurt users, is exactly the kind of tradeoff covered in observability and cost later in this roadmap — there is no universally correct answer, only a deliberate choice informed by your product’s actual traffic patterns and its tolerance for occasional slow responses during a genuine spike.