Linear regression

The simplest model that still teaches the whole idea

Linear regression predicts a continuous number as a weighted sum of the inputs plus a constant: a predicted house price as some amount per square foot, plus some amount per bedroom, plus a base value, each weighted by a coefficient the training process finds. It is usually the first model anyone learns, not because it is the most powerful but because nearly every idea that recurs through the rest of this roadmap — a cost function, fitting by optimisation, the risk of overfitting, interpreting what a model actually learned — shows up here in its simplest possible form.

The cost function: what “best fit” means

A line can be drawn through any scatter of points; the question is which line is best. Linear regression answers this with mean squared error: for every training point, compute the difference between the line’s prediction and the actual value, square that difference, and average over all points. Squaring matters for two reasons — it makes every error positive, so errors in opposite directions cannot cancel out and hide a genuinely bad fit, and it penalises a large error far more heavily than several small ones, which usually matches what “a bad prediction” should mean in practice.

Fitting the model means finding the coefficients that make this quantity as small as possible across the entire training set.

What to watch for

The video works a complete numeric example of this fitting process by hand — actual data points, the actual arithmetic of minimising squared error — which makes the cost-minimisation idea concrete in a way that a purely conceptual explanation leaves abstract. Watching the numbers move is worth more here than the equations alone.

Two ways to find the minimum

The normal equation solves for the optimal coefficients directly in one closed-form calculation, which works well when the number of input features is small. As the number of features grows, that direct calculation becomes too expensive, and gradient descent — covered in depth on the gradient descent formula page — instead finds the minimum iteratively: start from an arbitrary set of coefficients, compute which direction reduces the error fastest, take a small step in that direction, and repeat until the error stops improving meaningfully. Both methods arrive at the same answer for linear regression specifically; gradient descent is the one that generalises to the vastly more complex models, including every neural network, covered later in this roadmap.

Reading the coefficients

Once fit, each coefficient has a direct, literal meaning: holding every other input fixed, it is the change in the predicted output for a one-unit increase in that specific input. A coefficient of 3 on “years of experience” means one additional year of experience predicts three more units of whatever is being predicted, with every other input held constant. This interpretability is a genuine advantage linear regression retains over far more powerful models — a stakeholder can be shown exactly what the model concluded about each input, not merely how accurate its predictions turn out to be.

Where it breaks down

Linear regression assumes the relationship between inputs and output is genuinely linear, and it is not a safe assumption everywhere — a line is a poor fit to a relationship that curves, saturates, or interacts in ways a straight weighted sum cannot represent. It is also sensitive to outliers, because one far-flung point contributes a squared error disproportionate to how much it should actually influence the fit. These limits are exactly what motivate the regularization lesson that follows, and eventually the move toward models, including decision trees and neural networks, capable of representing relationships a straight line cannot.