How LLMs work

The one idea worth holding on to

A language model predicts the next token. Everything else — chat, reasoning, tool use, agents — is that single operation wrapped in enough scaffolding to look like something else. Keeping this in view explains behaviour that is otherwise baffling: why a model invents a citation with total confidence, why asking it to think step by step changes the answer, why the same prompt gives a different result twice.

From text to a distribution

A model’s input is a sequence of tokens — covered properly in the next lesson, but treat a token here as roughly a word fragment. Each token is converted to a vector by a lookup table called an embedding layer, so the sequence becomes a stack of vectors. Those vectors pass through dozens of repeated blocks, each combining two operations: self-attention, which lets every position in the sequence pull information from every earlier position, weighted by how relevant that earlier position is to this one, and a feed-forward network, which processes each position’s vector independently and is where most of a model’s raw parameter count actually lives. After the last block, a final linear layer projects the last position’s vector onto one score per word in the vocabulary — tens of thousands of scores — and a softmax turns those scores into a probability distribution that sums to one. That distribution is the model’s entire output for this step. Generating one token means sampling from it; generating the whole response means repeating the entire process, once per output token, with the newly produced token appended to the input each time.

Where the parameters came from

Those billions of parameters are not programmed; they are fit by gradient descent against a simple objective during pretraining: given everything so far, predict the next token correctly, scored against trillions of tokens of real text. Nothing in that objective asks the model to be honest, helpful or careful — only to predict plausible continuations. The behaviour that looks like following instructions or refusing harmful requests is added afterwards, in a second stage usually called instruction tuning or alignment, where the model is further trained on examples of the behaviour a product wants, and often refined again using human or automated feedback on which of several responses is better. The pretrained model is a raw continuation engine; the model you call through an API has had two or three additional rounds of training stacked on top to make it behave like an assistant.

Why generation is the expensive half

Processing the prompt happens in a single pass: every position attends to every earlier position at once, which is embarrassingly parallel on a GPU. Producing the response cannot be parallelised that way, because token five needs to exist before token six can be generated — each output token is its own full pass through every layer of the network, conditioned on everything produced so far. This asymmetry is the direct cause of the pricing split covered in cost and latency: reading is cheap because it is parallel, writing is expensive because it is sequential.

What “no memory” actually means

A deployed model holds no state between API calls. Every turn of what feels like a continuous conversation is, mechanically, a brand new forward pass over the entire transcript re-sent from scratch. The model is not “remembering” the earlier turns; it is reading them again, exactly as if for the first time, packed into the same context window as the current question. This is why a conversation that has grown very long costs more per turn even when the new message is short, and it is the reason the context window has to be managed as a budget rather than treated as free memory — the subject of a later lesson in this group.