Regularization

The problem regularization solves

A model with enough flexibility can fit its training data almost perfectly, including the noise specific to that one dataset — the overfitting problem covered in generalization and overfitting. One common symptom of this is unusually large coefficients: a model chasing noise often does so by leaning heavily on one or two inputs with large weights, producing predictions that swing wildly in response to small changes in those inputs. Regularization directly discourages this, by adding a penalty for coefficient size into the quantity the model is trained to minimise, alongside the usual prediction error.

What to watch for

The video covers both penalties together in one pass with worked examples — genuinely useful here, because this lesson needs both L1 and L2 plus the practical difference between them, and the most trusted channel on this exact topic splits that same material across a multi-part series rather than one video.

L2, or Ridge: shrink everything a little

L2 regularization adds a penalty proportional to the sum of the squared coefficients. Minimising prediction error plus this penalty together pushes every coefficient smaller than it would otherwise be, smoothly, without driving any of them exactly to zero. This produces a model that relies a little less heavily on any single input, trading a small amount of fit on the training data for a model expected to generalise better to new data.

L1, or Lasso: shrink some all the way to zero

L1 regularization adds a penalty proportional to the sum of the absolute values of the coefficients, rather than their squares. The practical consequence is different in kind, not just degree: L1’s penalty can push a coefficient all the way to exactly zero, effectively removing that feature from the model entirely. Applied to a dataset with many candidate features, L1 regularization therefore acts as automatic feature selection — the features that survive with a non-zero coefficient are the ones the model found genuinely useful, and the rest are cut out entirely rather than merely shrunk.

Choosing between them, and choosing the strength

Prefer L1 when a dataset has many candidate features and a genuine suspicion that only some of them matter — the automatic feature selection is directly valuable there. Prefer L2 when most features are at least somewhat informative and the goal is simply a smoother, less overfit model rather than a sparser one. Some approaches, often called elastic net, combine both penalties at once rather than forcing a choice between them.

Either way, the strength of the penalty — how heavily large coefficients are punished — is itself a setting the model does not learn on its own; it has to be chosen, typically by trying several strengths and checking which performs best on held-out data, exactly the kind of decision cross-validation and hyperparameter tuning are built to make systematically. Too little penalty barely addresses overfitting at all; too much shrinks away coefficients that were genuinely useful, trading overfitting for underfitting instead of finding the balance between them.