Embeddings

Meaning as a position

An embedding model takes text and returns a fixed-length list of numbers — typically a few hundred to a few thousand dimensions — chosen so that the geometry carries semantics. Texts about the same thing end up pointing in the same direction, and that single property is what makes search over meaning rather than keywords possible. Nothing about this is magic: the model is trained so that pairs of texts known to be related (a question and its matching answer, a sentence and its paraphrase, two chunks from the same document) are pushed closer together in the vector space, while unrelated pairs are pushed apart. This is usually done with a contrastive training objective — the model sees a text, a genuinely matching text, and several unrelated texts, and is adjusted so the match scores highest among them. Doing this over millions of such triplets is what shapes the geometry into something useful.

What the numbers actually look like

A modern embedding model might return 768, 1536 or 3072 numbers per piece of text, and none of those individual numbers means anything on its own — there is no dimension that corresponds to “is about finance” or “is a question.” Meaning lives only in the relationship between whole vectors, which is why embeddings are always compared to each other and never read directly. Most APIs normalise the output to unit length, so every vector lies on the surface of a high-dimensional sphere; this is what makes cosine similarity and the raw dot product interchangeable in practice, a point the similarity search lesson relies on.

Why closeness is measured by angle

Two vectors pointing in nearly the same direction are considered similar regardless of how long either one is, because length mostly reflects incidental properties like text length or word frequency rather than meaning. A short, punchy sentence and a long, hedging paragraph that both express the same idea should score as similar — and they do, because the comparison discards magnitude and keeps only direction. This is also precisely why an embedding can be confidently, uselessly wrong: “the drug is safe at this dose” and “the drug is not safe at this dose” are about the same medication, the same dose and the same property, so they point in almost the same direction and score as highly similar, even though they assert opposite things. An embedding encodes topic and framing, not truth value.

Newer techniques worth knowing

Two developments are common enough now to mention even though neither appears in the video. Matryoshka embeddings are trained so that a prefix of the full vector — say, the first 256 of 1536 numbers — is still a valid, useful embedding on its own, just a less precise one. This lets an application store a short vector for a fast initial pass over a huge corpus and only use the full vector where it matters, trading a little quality for a large reduction in storage and compute. Late-interaction models, by contrast, keep a separate vector per token rather than collapsing a whole passage into one vector, and compare query and document token by token at search time — more expensive, often noticeably more accurate on short, keyword-dense queries. Neither approach changes the one-vector-per-document story in this lesson so much as it refines it, and both are worth knowing the names of before you assume “an embedding” always means one fixed vector.

What embeddings are not for

They are not a fact store, not a ranking of quality, and not a substitute for exact matching. A product code, a date or a person’s name is better found by the keyword half of a hybrid search — which is why similarity search does not rely on vectors alone. And because an embedding model is itself a fixed, versioned artifact, re-embedding your whole corpus is required whenever you switch models — vectors from two different embedding models are not comparable to each other at all, even if they happen to have the same number of dimensions.