Multimodal AI

One representation space

The trick that makes multimodal models work at all is mechanically unglamorous: convert every input type into vectors that live in the same representation space the text tokens already occupy, then let the usual transformer machinery — covered in how LLMs work — run over all of them together without needing to know which modality any particular vector originally came from. An image is cut into a grid of small patches, and each patch is projected by a trained encoder into a vector the same size as a text token’s embedding. Audio is processed into acoustic features over short time windows and similarly projected. Video is sampled at some chosen rate into individual frames and treated as a sequence of images with an added time dimension. Once this projection has happened, the model attends across modalities essentially for free, since by the time self-attention runs, every single position in the sequence is simply a vector of the same shape as every other, regardless of whether it originated as a word, an image patch, or a moment of audio.

Concrete token costs, so the budget is not a surprise

Vision tokens scale with image resolution in a way that catches people off guard the first time they see a real bill. A low-resolution image might cost on the order of a few hundred tokens once converted to patches, while a high-resolution image can cost several thousand — easily as much as multiple pages of dense text — purely because of how finely it was divided into patches before being handed to the encoder. Video compounds this directly: each sampled frame costs roughly as much as a single still image, so a short video sampled at even a modest rate of one frame per second can consume tens of thousands of tokens within a span of just a few minutes, which is often enough to exhaust a large fraction of an entire context window on video content alone, with comparatively little budget left for anything else in the prompt. Downscaling images before upload and deliberately sampling fewer frames per second are the two most direct levers for controlling this cost, and both are worth treating as a tuning parameter rather than leaving at whatever a library’s convenient default happens to be.

The voice latency chain, and where it actually breaks

A conversational voice agent built the traditional way chains three separate stages: speech-to-text converts the user’s spoken audio into text, that text is sent through the ordinary LLM pipeline exactly as if it had been typed, and text-to-speech converts the resulting response back into audio for playback. Each stage in this chain adds its own latency on top of the others, and the sum of all three is what the user actually experiences as the pause before a response begins. Newer native audio-to-audio models instead process spoken input and produce spoken output directly within a single model, entirely skipping the intermediate text representation and its two separate conversion stages — meaningfully reducing the end-to-end latency for this specific use case, at the cost of being a considerably newer and less thoroughly battle-tested technology than the well-established three-stage chain, with fewer provider options and less production track record to draw on when something goes wrong.

Vision understanding versus OCR, as competing approaches

For extracting information from documents — invoices, scanned forms, tables embedded in images — there are genuinely two competing approaches, not one obvious answer. Traditional OCR (optical character recognition) converts an image directly into plain text with comparatively high precision on exactly the text content it recognises, but with essentially no understanding of layout, meaning, or visual structure beyond the bare sequence of characters it extracted. A vision-capable language model can instead reason directly over the image — correctly understanding that a particular number sits specifically inside the “total due” cell of a table rather than some other cell — but may occasionally misread an individual character with more confidence than a dedicated OCR engine would show for that same character. In practice, many production document-processing pipelines combine both: OCR extracts raw, reliable text as a verification cross-check, while a vision model separately reasons over structure and overall layout, with results from each reconciled against the other before anything downstream treats the extracted data as trustworthy.

Multimodal RAG: embedding images for retrieval

The retrieval techniques covered earlier in this roadmap extend directly to images once a suitable embedding model exists for them: an image can be embedded into the same kind of vector space a text chunk occupies, and a query — either text or another image — retrieves the most semantically similar images from an indexed corpus using exactly the same underlying similarity-search machinery covered in similarity search. This becomes useful well beyond image search as a standalone feature: a product catalogue can be searched by a photo rather than only by a typed description, or a large corpus of scanned, image-based documents can be made searchable without first needing a perfect, fully reliable text extraction of every single page.

Why this remains the thinnest node in the roadmap

No single available source covers vision understanding, speech recognition and synthesis in both directions, and generation together in one coherent treatment, and speech specifically has essentially no good general explainer available anywhere. Evaluation is also measurably harder here than for text: comparing two candidate transcriptions or two different descriptions of the same image is not simple string equality the way checking many text outputs can be, so eval design for a multimodal feature genuinely needs more careful, deliberate thought than anywhere else covered in this roadmap, often relying on an LLM-as-judge approach adapted specifically to compare meaning rather than exact wording.