Evaluation metrics

Why a single accuracy number is not enough

As the imbalanced data lesson established, a model can post a very high accuracy while being nearly useless on the class that actually matters. Evaluation metrics exist precisely to look past that single aggregate number and ask more specific, more honest questions about what a model actually gets right and wrong.

The confusion matrix: where everything else starts

Every classification outcome falls into exactly one of four categories. A true positive is a correct positive prediction; a true negative is a correct negative prediction. A false positive is predicting positive when the truth was negative — a false alarm. A false negative is predicting negative when the truth was positive — a miss. The confusion matrix is simply a table of how many predictions fell into each of these four categories, and nearly every classification metric that follows is computed directly from these four counts.

What to watch for

The video builds exactly this matrix from scratch and ties it directly to why accuracy alone can mislead — the foundation everything else in this lesson is built on top of.

Precision and recall: two different questions

Precision asks: of everything the model predicted as positive, what fraction actually was? It is true positives divided by the sum of true positives and false positives — a measure of how trustworthy a positive prediction is. Recall asks a different question: of everything that was actually positive, what fraction did the model correctly find? It is true positives divided by the sum of true positives and false negatives — a measure of how thorough the model is at catching real positives.

These two typically trade against each other. A model can achieve perfect recall trivially by predicting positive for everything, at the cost of terrible precision, since it would also flag every true negative as a false positive. It can achieve high precision by predicting positive only when extremely confident, at the cost of missing many true positives and so lowering recall. The F1 score combines both into one number — their harmonic mean — useful as a single summary when both matter and neither should be allowed to be very weak while the other is strong, though it hides exactly which of the two is doing the work, which is why looking at precision and recall separately usually tells you more than the combined score alone.

Which one to prioritize depends on the cost of each mistake

There is no universally correct choice between prioritising precision or recall — it depends entirely on which kind of mistake costs more in the specific application. A missed medical diagnosis is usually far more costly than an unnecessary follow-up test, arguing for prioritising recall even at the expense of precision. An automated system that triggers an expensive manual review on every flagged case needs its false-alarm rate controlled carefully, arguing for prioritising precision instead. This decision connects directly back to the decision-threshold choice covered in logistic regression — moving that threshold is exactly how precision and recall get traded against each other in practice.

Regression metrics: measuring continuous error

For predicting a continuous value rather than a category, mean squared error (MSE) averages the squared difference between predictions and actual values — the same quantity linear regression is trained to minimise, covered in linear regression. RMSE, its square root, converts that error back into the original units of the outcome, which makes it considerably easier to interpret directly — an RMSE of five dollars on a price-prediction problem is immediately meaningful in a way a squared-dollars number is not. MAE (mean absolute error) averages the plain absolute difference instead of the squared one, which penalises large errors less severely than MSE or RMSE do.

R² answers a different question entirely: what fraction of the variance in the outcome does the model actually explain, on a scale from zero (explains none of it, no better than always predicting the average) to one (explains all of it perfectly)? Because it is a standardised fraction rather than a raw error in the outcome’s own units, R² is comparable across entirely different problems in a way that RMSE, tied to whatever units the original outcome happens to be measured in, simply is not.