Embedding and multimodal inference

A genuinely different shape of workload

Every lesson in this roadmap so far has assumed the shape of autoregressive text generation: a prefill pass over the prompt, followed by a sequential decode loop producing one token after another. Embedding models and vision-language models do not share this shape, and serving them well means recognising exactly how they differ rather than assuming the same techniques apply unchanged.

Embedding models: prefill, and nothing else

An embedding model takes a piece of text (or an image, for some models) and produces a single fixed-size vector in one forward pass — there is no decode phase at all, because there is nothing to generate token by token; the entire output already exists once that one pass completes. This makes an embedding model’s performance profile much closer to prefill alone, as covered in prefill and decode — largely compute-bound, highly parallelisable, with none of the sequential, memory-bandwidth-bound decode loop that dominates the cost of serving a text-generation model. Batching many embedding requests together is correspondingly simpler and more effective than batching generation requests, precisely because there is no variable-length decode loop to coordinate across the batch.

What to watch for

No video dedicated specifically to serving embedding models or vision-language models exists on YouTube, after searching seven distinct angles — this is a genuine gap in available material, not a missed search. This lesson instead reuses a substantial architecture talk already in this roadmap’s serving software group: Ion Stoica walks through how vLLM and SGLang are actually built, and the same engine families covered there are what serve embedding models and VLMs in production, configured differently for each workload’s specific shape. The embedding- and multimodal-specific content in this lesson is carried entirely in this written material, not in the video.

Vision-language models: an encoding step before any text

A vision-language model (VLM) processes an image or video alongside text, and before any of the text pipeline runs at all, the visual input has to be encoded into a form the language model component can actually consume — a separate compute step with its own real cost. In practice this commonly means an image contributes a large number of effective tokens to the prompt’s budget, sometimes more than a substantial amount of accompanying text would, which has to be accounted for in exactly the way the context window is budgeted for a purely text-based request. A video adds a further dimension: frames sampled over time each carry this same encoding cost, so the sampling rate chosen becomes a direct and significant cost and latency lever, in exactly the way covered for multimodal inputs in Upshard’s AI Engineer roadmap.

Why generic text-serving infrastructure does not automatically fit

An inference engine tuned purely around the assumption of a prefill-then- decode text workload can handle embedding and multimodal requests, but not necessarily well: an embedding workload gains little from decode-oriented optimisations like speculative decoding, since there is no decode loop to speed up at all, and a vision-language workload’s image-encoding step is a cost category the usual prefill/decode accounting does not capture on its own. Recognising that these are genuinely different workload shapes, rather than minor variations on text generation, is the first step toward provisioning and tuning serving infrastructure that actually fits them.