Shannon entropy
The average surprise in a random variable, and the floor on how few bits any lossless code can spend on it. A fair coin costs one bit; a coin that always lands heads costs nothing to describe.
- The random variable being measured.
- Its probability mass function.
- Base two for bits, base e for nats.