Open vs closed models

Someone else’s inference engineering, or your own

Calling a closed model through an API means every lesson in this roadmap has already been handled by someone else — the provider chose the hardware, built the serving stack, tuned the batching, and absorbed the operational burden of keeping it running. You get a simple interface and none of the control: you cannot change the hardware, cannot tune the serving engine, and pay whatever the provider’s pricing says per token.

Self-hosting an open-weight model inverts this completely. You choose the GPUs, pick or build the serving engine, own the scaling and the on-call rotation — and in exchange you control every lever this roadmap covers: cost per request, the exact latency profile, data residency, and the freedom to fine-tune or modify the model itself.

What to watch for

Fireworks AI’s CTO walks through this exact tradeoff directly: calling closed frontier APIs gets you a fast start with no infrastructure, but real cost, latency and customisation limits; self-hosting buys full control at the cost of real operational burden. It is a vendor talk — Fireworks sells a managed middle ground between the two extremes — so it understandably frames pure self-hosting’s burden on the heavier side. The underlying tradeoff it describes is accurate regardless.

The decision is rarely all-or-nothing

In practice, most real products do not pick one side permanently. A team might call a closed API for most traffic, where the convenience outweighs the cost, and self-host a smaller open-weight model for one high-volume, latency-sensitive path where the economics of self-hosting clearly win once volume is high enough. A company handling regulated data might be required to self-host regardless of cost, because sending that data to a third-party API is simply not permitted. The right split is an economic and operational calculation specific to the product, not a one-time philosophical choice.

A rough rule of thumb for the crossover

Where volume is genuinely the deciding factor, a useful way to think about it is comparing the per-token price a closed API charges against the fully loaded cost of a GPU — purchase or rental price amortised over its useful life, plus power and the engineering time to operate it — divided by how many tokens that GPU can actually serve over the same period. At low volume, the GPU sits partly idle between requests and loses this comparison badly, since you are paying for capacity you are not using. At high, sustained volume, the GPU’s fixed cost gets spread across enough tokens that it wins decisively, often by a wide margin, because the API’s per-token price has to include the provider’s own margin and their own amortised infrastructure cost on top of the same underlying hardware economics. Where exactly this crossover sits depends on the model size, the hardware chosen, and how close to constantly utilised the self-hosted capacity actually runs — which is why it is worth calculating for your own real traffic rather than assumed from a rule someone else quotes.

What this roadmap assumes from here

Every lesson that follows — hardware, serving engines, quantisation, batching, scaling across GPUs — matters only once you have chosen, for at least part of your traffic, to run inference yourself. If you call a closed API exclusively, most of this roadmap describes work someone else is already doing on your behalf. The value in understanding it anyway is knowing what that provider is actually doing for the price you pay, and recognising the exact moment, if it ever comes, when owning that work yourself would serve your product better.