Speech inference

A latency budget tighter than almost anything else in this roadmap

Every lesson so far has treated latency as something to minimise within whatever a product reasonably tolerates. Real-time voice conversation sets a specific, unforgiving bar: a response that takes much longer than a few hundred milliseconds to begin breaks the feeling of a natural conversation, in a way a slightly slow chat response simply does not. Serving speech models well means hitting this bar consistently, which shapes the architecture choices covered in this lesson more than almost any other constraint in this roadmap.

The chained pipeline, and why every stage has to stream

The traditional architecture chains three separate models: automatic speech recognition (ASR) converts spoken audio into text, a language model processes that text and produces a response, and text-to-speech (TTS) converts the response back into spoken audio. Run naively — wait for ASR to fully finish before starting the language model, wait for the language model to fully finish before starting TTS — the latencies of all three stages stack sequentially, which makes hitting a sub-second total budget essentially impossible. The fix is streaming every stage: ASR starts emitting partial transcriptions before the speaker has finished talking, the language model can begin generating a response from an early partial transcript, and TTS begins synthesising audio from the first part of the response before the rest has even been generated. This overlapping, rather than sequential, execution is what makes a tight conversational budget achievable with a three-stage chain at all.

What to watch for

A practitioner building exactly this in production walks through orchestrating streaming speech-to-text, language model reasoning, and text-to-speech against a real conversational latency budget — targeting roughly a hundred milliseconds for transcription completion alone — which is the concrete engineering this lesson is about, not just the architectural idea in the abstract.

Native speech-to-speech: removing the stages entirely

An alternative architecture processes audio directly end to end, with no intermediate text representation at all — the model takes spoken audio as input and produces spoken audio as output, skipping the ASR and TTS conversion stages and the handoffs between three separate models entirely. This removes real latency cost by construction, since there are no stage transitions left to pay for, but it is a newer approach with less production track record than the well-established chained architecture, and it trades away the ability to easily inspect or modify the intermediate text representation, which a chained pipeline gets for free as a natural byproduct of its architecture. The deeper link covers this exact contrast directly, comparing the chained approach against a native speech-to-speech implementation built on a realtime API.

Where this sits relative to the rest of this roadmap

Every technique covered earlier still applies to each individual model in either architecture — a speech-to-text model still benefits from quantization and a well-chosen inference engine, and a chained pipeline’s overall latency is still bounded by time to first token at each individual stage. What is specific to this lesson is the orchestration across stages and the unusually tight end-to-end budget that forces streaming everywhere, rather than any single model’s serving mechanics being fundamentally different from what the rest of this roadmap covers.