Top-p (nucleus) sampling

upshard

\mathcal{V}_p = \min \left\{ \mathcal{V}' \subseteq \mathcal{V} : \sum_{x \in \mathcal{V}'} p(x) \geq p \right\}

Keeps the smallest set of tokens whose probabilities already add up to p, then samples within it. The cutoff moves with the model's confidence, which is what makes it behave better than a fixed top-k.

Symbols

$\mathcal{V}$: The full vocabulary.
$\mathcal{V}_p$: The nucleus — the kept set.
$p$: The cumulative probability mass to retain, often 0.9.

Save your progress

Your account

Delete your account?

Top-p (nucleus) sampling

Keyboard shortcuts