Hallucinations and limitations

Fluency is not a truth signal

A hallucination is a model stating something false — a fabricated citation, a wrong date, a function that doesn’t exist in a library — with the same confident, grammatically correct delivery it uses for true statements. This follows directly from how generation works, covered in the how-LLMs-work lesson: the model predicts the next token based on what’s statistically plausible given everything before it, not based on verifying claims against an external source of truth. Fluency and correctness are produced by the same mechanism, so one gives you no information about the other. A model can be maximally confident-sounding while being completely wrong, because “sounding confident” and “being right” are not linked inside its generation process the way they’d need to be for confidence to mean anything.

Hallucinations show up in recognizable patterns. Fabricated specifics — invented citations, URLs, statistics, or API functions that sound exactly like real ones because they’re built from the same patterns as the real article. Overconfident extrapolation — asked something slightly outside its knowledge, the model extends a pattern rather than flagging the gap. Self-contradiction across a long response — without persistent working memory beyond what’s in the current context, a model can state one fact early and an incompatible one later, because each token is generated from local plausibility rather than a check against everything said earlier. Knowing these patterns matters practically: the later lesson on evaluation and testing covers how to systematically catch them before they reach a user, which is the real mitigation — not hoping a model volunteers its own uncertainty.

The knowledge cutoff is a hard wall

Every model is trained on a dataset collected up to some date, after which it has no information at all — not “weaker” information, none. Ask about something after that date and a well-behaved model might say it doesn’t know, but it might instead pattern-match to what similar information usually looks like and generate something plausible-sounding and wrong. This is a structural limitation, not a bug that improves with a better prompt: no amount of clever phrasing gives a model access to facts it was never trained on. The practical fix lives outside the prompt itself — retrieval systems that fetch current information and insert it into the context window, which is exactly why context windows (previous lesson) and retrieval exist as part of real systems rather than relying on the model’s frozen training knowledge alone.

No memory beyond what’s in front of it

A model has no persistent memory across separate calls unless the surrounding application builds that back in by re-sending prior conversation as part of the context window. Each API call is stateless: the model sees exactly the tokens you send it in that request and nothing else — not what you discussed yesterday, not what a different user asked five minutes ago. Products that feel like they “remember” you are doing exactly that work — storing conversation history and re-injecting relevant parts of it into each new prompt — and that re-injected history still has to fit inside the same finite context window covered in the previous lesson. This is worth internalizing early: if a model seems to have forgotten something, it usually hasn’t forgotten in any meaningful sense — the information simply isn’t in the current prompt, often because it fell outside the context window or because the application layer didn’t carry it forward.

Where reasoning actually breaks down

Models are strongest on tasks that resemble patterns seen often in training: fluent writing, common code idioms, well-known facts. They’re measurably weaker on exact computation (multiplying large numbers reliably, for instance, where a single wrong digit compounds with no self-correction), long deterministic multi-step logic where an early error propagates silently through every later step, and genuinely novel problems that don’t resemble anything in the training distribution. None of this means the model is “bad at reasoning” in some vague sense — it means the underlying mechanism is pattern continuation, and tasks that require exact, verifiable, step-by-step correctness are a worse fit for that mechanism than tasks that reward fluent, plausible generation. Later lessons on chain-of-thought and self-consistency prompting cover techniques that measurably help here, precisely because they change what pattern the model is continuing — but they manage the limitation, they don’t remove it.

Designing around all of this

The practical takeaway isn’t pessimism about what models can do — it’s building workflows that account for these limits rather than assuming they don’t exist. Don’t trust an unverified factual claim from a model on anything consequential; verify or cite. Don’t expect knowledge of recent events without retrieval. Don’t expect persistent memory without explicit engineering to provide it. And for tasks demanding exact, multi-step correctness, treat the model’s output as a strong first draft that needs checking, not a guaranteed-correct final answer. Every later lesson in this roadmap — on structured output, on evaluation, on reliability — is in some sense a technique for managing one of these limitations well.