Kullback–Leibler divergence

upshard

D_{\mathrm{KL}}(P \parallel Q) = \sum_{x} P(x) \log \frac{P(x)}{Q(x)}

The extra bits you pay, on average, when you encode samples drawn from P using a code built for Q. It is zero only when the two distributions agree everywhere, and it is not symmetric — swapping P and Q gives a different number.

Symbols

$P$: The true distribution — the one the data actually comes from.
$Q$: The approximating distribution — the model, or the code you built.

Save your progress

Your account

Delete your account?

Kullback–Leibler divergence

Keyboard shortcuts