Time to first token

The number a user actually waits on

Time to first token is exactly what it sounds like: the elapsed time between sending a request and seeing the first piece of the response appear. In a streaming chat interface, this is the length of the blank, silent pause before anything shows up at all — and it is the single latency number most directly tied to how fast a product feels, independent of how long the full response eventually takes to finish streaming.

Why prefill dominates it

Generating a response has two phases, covered properly in the next lesson: prefill, which processes the entire input prompt in one parallel pass, and decode, which produces output tokens one at a time afterward. The first output token cannot appear until prefill has finished processing the whole prompt — there is no way to start generating before the model has seen everything it was given. This means TTFT is, to a very good approximation, the time prefill takes, and prefill’s cost scales directly with how many input tokens there are to process.

What to watch for

Mark Moyou walks through tokenization, prefill and decode, and defines TTFT explicitly as the time to the first output token, tying it directly to the cost of processing the whole prompt before anything can be produced — exactly this lesson’s scope. The embed is trimmed to the portion of the talk that covers this directly; the fuller talk continues into broader inference economics covered elsewhere in this roadmap.

Why this means your prompt length is a latency decision

A system prompt padded with unnecessary instructions, or a retrieval step that returns more context than the question actually needs, is not merely a cost decision — it is a direct tax on every single request’s TTFT, because every one of those extra tokens has to be processed in prefill before the user sees anything. This is the same economic pressure that makes prompt caching worth having at the serving level: a cached prefix skips re-processing exactly the part of prefill that does not change between requests, which is a direct TTFT win and not merely a cost one.

A worked sense of the numbers

Prefill cost scales with prompt length in a way that is worth seeing concretely. A short, ten-token question might take a few tens of milliseconds to process on a well-provisioned GPU. A prompt padded with a lengthy system message, several retrieved documents and a long conversation history — easily several thousand tokens — can take an order of magnitude longer to prefill, simply because there is that much more to process before the first output token can appear. This is why a product with a verbose, unoptimised system prompt can feel noticeably slower to start responding than one with a lean prompt, even when both are calling the identical model on identical hardware — the difference is entirely in how much prefill work each one demands before anything becomes visible.

TTFT is not the whole latency story

A fast TTFT with slow subsequent generation still leaves a user waiting a long time for the complete answer — TTFT is necessary for a product to feel responsive, not sufficient for it to actually finish quickly. The next two lessons, on throughput and the tradeoff between latency and throughput, cover the rest of what determines how fast a response actually completes once it has started.