Formula library

The maths the lessons lean on. Each entry is written out, glossed in plain English, has its symbols named, and links to the topics that use it.

Name	Expression	Domains	Lessons
Action-value function	$Q^{\pi}(s, a) = \mathbb{E}_{\pi}\!\left[G_t \mid S_t = s, A_t = a\right]$	Reinforcement learning	1
Adam	$\theta_{t+1} = \theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$	Optimization, Deep learning	2
Advantage function	$A^{\pi}(s, a) = Q^{\pi}(s, a) - V^{\pi}(s)$	Reinforcement learning	—
Arithmetic intensity	$I = \frac{\text{FLOPs}}{\text{bytes moved}}$	Inference	4
Batch normalization	$\mathrm{BN}(x_i) = \gamma \frac{x_i - \mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}} + \beta$	Deep learning	1
Bayes' theorem	$P(A \mid B) = \frac{P(B \mid A)\, P(A)}{P(B)}$	Probability	—
Bellman expectation equation	$V^{\pi}(s) = \mathbb{E}_{\pi}\!\left[r + \gamma V^{\pi}(s') \mid s\right]$	Reinforcement learning	1
Bellman optimality equation	$V^{}(s) = \max_{a} \mathbb{E}\!\left[r + \gamma V^{}(s') \mid s, a\right]$	Reinforcement learning	—
Bernoulli distribution	$p(x) = \theta^{x} (1-\theta)^{1-x}, \quad x \in \{0, 1\}$	Probability	—
Binomial distribution	$P(k) = \binom{n}{k} \theta^{k} (1-\theta)^{n-k}$	Probability	—
Central limit theorem	$\frac{\bar{X}_n - \mu}{\sigma / \sqrt{n}} \xrightarrow{d} \mathcal{N}(0, 1)$	Probability	—
Chain rule	$\frac{d}{dx} f(g(x)) = f'(g(x)) \cdot g'(x)$	Calculus	1
Channel capacity	$C = \max_{p(x)} I(X; Y)$	Information theory	—
Chinchilla scaling law	$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$	Inference	1
Conditional entropy	$H(Y \mid X) = H(X, Y) - H(X)$	Information theory	—
Conditional probability	$P(A \mid B) = \frac{P(A \cap B)}{P(B)}$	Probability	—
Cosine learning-rate schedule	$\eta_t = \eta_{\min} + \tfrac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\frac{t\pi}{T}\right)$	Optimization	—
Cosine similarity	$\operatorname{sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\\|\mathbf{a}\\|\,\\|\mathbf{b}\\|}$	Linear algebra	4
Covariance	$\operatorname{Cov}(X, Y) = \mathbb{E}\!\left[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])\right]$	Probability	1
Cross-entropy	$H(P, Q) = -\sum_{x} P(x) \log Q(x)$	Information theory, Deep learning	3
Derivative	$f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$	Calculus	—
Determinant (2 by 2)	$\det\begin{pmatrix} a & b \\ c & d \end{pmatrix} = ad - bc$	Linear algebra	—
Directional derivative	$D_{\mathbf{u}} f = \nabla f \cdot \mathbf{u}$	Calculus	—
Discounted return	$G_t = \sum_{k=0}^{\infty} \gamma^{k} r_{t+k+1}$	Reinforcement learning	1
Discrete convolution	$(f * g)[n] = \sum_{m} f[m]\, g[n-m]$	Deep learning	1
Dot product	$\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_i b_i = \\|\mathbf{a}\\|\,\\|\mathbf{b}\\|\cos\theta$	Linear algebra	3
Dropout	$\tilde{h} = \frac{m \odot h}{1 - p}, \qquad m_i \sim \mathrm{Bernoulli}(1-p)$	Deep learning	1
Eigenvalue equation	$A\mathbf{v} = \lambda \mathbf{v}, \quad \mathbf{v} \neq \mathbf{0}$	Linear algebra	1
Euclidean norm	$\\|\mathbf{x}\\|_2 = \sqrt{\sum_{i=1}^{n} x_i^2}$	Linear algebra	4
Expected value	$\mathbb{E}[X] = \sum_{x} x\, p(x)$	Probability	1
Fundamental theorem of calculus	$\int_a^b f'(x)\, dx = f(b) - f(a)$	Calculus	—
GELU	$\mathrm{GELU}(x) = x \cdot \Phi(x)$	Deep learning	1
Gradient	$\nabla f = \left( \frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n} \right)$	Calculus	1
Gradient descent	$\theta_{t+1} = \theta_t - \eta \nabla_{\theta} L(\theta_t)$	Optimization	2
Hessian	$H_{ij} = \frac{\partial^2 f}{\partial x_i \, \partial x_j}$	Calculus	—
Integration by parts	$\int u \, dv = uv - \int v \, du$	Calculus	—
Jacobian	$J_{ij} = \frac{\partial f_i}{\partial x_j}$	Calculus	—
Jensen–Shannon divergence	$\mathrm{JSD}(P \parallel Q) = \tfrac{1}{2} D_{\mathrm{KL}}(P \parallel M) + \tfrac{1}{2} D_{\mathrm{KL}}(Q \parallel M)$	Information theory	—
Joint entropy	$H(X, Y) = -\sum_{x, y} p(x, y) \log p(x, y)$	Information theory	—
Kullback–Leibler divergence	$D_{\mathrm{KL}}(P \parallel Q) = \sum_{x} P(x) \log \frac{P(x)}{Q(x)}$	Information theory, Probability	—
KV-cache size	$M_{\text{KV}} = 2 \cdot L \cdot H \cdot d_h \cdot S \cdot B \cdot b$	Inference	8
L1 regularization	$L_{\text{total}} = L + \lambda \sum_{i} \|\theta_i\|$	Optimization	1
L2 regularization	$L_{\text{total}} = L + \lambda \\|\theta\\|_2^2$	Optimization	1
Lagrange multipliers	$\mathcal{L}(x, \lambda) = f(x) - \lambda\, g(x)$	Optimization	—
Law of total probability	$P(A) = \sum_{i} P(A \mid B_i)\, P(B_i)$	Probability	—
Layer normalization	$\mathrm{LN}(\mathbf{x}) = \gamma \odot \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$	Deep learning	—
Little's law	$L = \lambda W$	Inference	5
Matrix multiplication	$(AB)_{ij} = \sum_{k=1}^{n} A_{ik} B_{kj}$	Linear algebra	—
Memory-bound token latency	$t_{\text{token}} \approx \frac{M_{\text{weights}} + M_{\text{KV}}}{\mathrm{BW}}$	Inference	4
Model memory footprint	$M_{\text{weights}} = N \cdot b$	Inference	7
Momentum	$v_{t+1} = \beta v_t + \nabla_{\theta} L, \qquad \theta_{t+1} = \theta_t - \eta v_{t+1}$	Optimization	1
Mutual information	$I(X; Y) = \sum_{x, y} p(x, y) \log \frac{p(x, y)}{p(x)\, p(y)}$	Information theory	—
Newton's method	$\theta_{t+1} = \theta_t - H^{-1} \nabla_{\theta} L$	Optimization	—
Normal distribution	$p(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$	Probability	1
Outer product	$(\mathbf{u} \mathbf{v}^{\top})_{ij} = u_i v_j$	Linear algebra	—
Perplexity	$\mathrm{PPL} = \exp\!\left(-\frac{1}{N} \sum_{i=1}^{N} \log p(x_i \mid x_{<i})\right)$	Information theory, Inference	2
PPO clipped objective	$L = \mathbb{E}\!\left[\min\!\left(\rho_t A_t, \; \mathrm{clip}(\rho_t, 1-\epsilon, 1+\epsilon) A_t\right)\right]$	Reinforcement learning	—
Product rule	$(fg)' = f'g + fg'$	Calculus	—
Q-learning update	$Q(s, a) \leftarrow Q(s, a) + \alpha\!\left[r + \gamma \max_{a'} Q(s', a') - Q(s, a)\right]$	Reinforcement learning	1
REINFORCE policy gradient	$\nabla_{\theta} J = \mathbb{E}_{\pi_\theta}\!\left[\nabla_{\theta} \log \pi_{\theta}(a \mid s)\, G_t\right]$	Reinforcement learning	—
ReLU	$\mathrm{ReLU}(x) = \max(0, x)$	Deep learning	2
RMSProp	$s_{t+1} = \rho s_t + (1-\rho)(\nabla_\theta L)^2, \qquad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{s_{t+1}} + \epsilon} \nabla_\theta L$	Optimization	1
Scaled dot-product attention	$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right) V$	Deep learning	4
Self-information	$I(x) = -\log p(x)$	Information theory	—
Shannon entropy	$H(X) = -\sum_{x} p(x) \log p(x)$	Information theory, Probability	1
Sigmoid	$\sigma(x) = \frac{1}{1 + e^{-x}}$	Deep learning	3
Singular value decomposition	$A = U \Sigma V^{\top}$	Linear algebra	1
Sinusoidal positional encoding	$PE_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right)$	Deep learning	1
Softmax	$\sigma(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$	Deep learning, Probability	5
Speculative decoding speedup	$\mathbb{E}[\text{tokens per step}] = \frac{1 - \alpha^{k+1}}{1 - \alpha}$	Inference	—
State-value function	$V^{\pi}(s) = \mathbb{E}_{\pi}\!\left[G_t \mid S_t = s\right]$	Reinforcement learning	1
Stochastic gradient descent	$\theta_{t+1} = \theta_t - \eta \nabla_{\theta} L_{\mathcal{B}}(\theta_t)$	Optimization	2
Taylor series	$f(x) = \sum_{n=0}^{\infty} \frac{f^{(n)}(a)}{n!} (x-a)^n$	Calculus	—
Temperature sampling	$p_i = \frac{\exp(z_i / T)}{\sum_{j} \exp(z_j / T)}$	Inference, Deep learning	3
Temporal-difference error	$\delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)$	Reinforcement learning	—
Top-p (nucleus) sampling	$\mathcal{V}_p = \min \left\{ \mathcal{V}' \subseteq \mathcal{V} : \sum_{x \in \mathcal{V}'} p(x) \geq p \right\}$	Inference	3
Trace	$\operatorname{tr}(A) = \sum_{i=1}^{n} A_{ii}$	Linear algebra	—
Transformer forward FLOPs	$C_{\text{fwd}} \approx 2ND$	Inference	2
Variance	$\operatorname{Var}(X) = \mathbb{E}\!\left[(X - \mathbb{E}[X])^2\right]$	Probability	2
Vector projection	$\operatorname{proj}_{\mathbf{b}}\mathbf{a} = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{b} \cdot \mathbf{b}} \mathbf{b}$	Linear algebra	—

Save your progress

Your account

Delete your account?

Formula library

Keyboard shortcuts