K-nearest neighbors

Predicting by resemblance

K-nearest neighbors is built on an idea simpler than almost anything else in this roadmap: to predict a new point, find the k training examples closest to it and predict based on what they show. For classification, that usually means a majority vote among the k neighbours’ labels; for predicting a continuous value, it usually means their average. There is no formula fit during training at all — the “model” is essentially just the stored training data itself, which is why this approach is called non-parametric: it makes no assumption in advance about the shape of the relationship between inputs and output, unlike linear regression’s assumption that the relationship is a straight line.

What to watch for

The video covers the core mechanism directly — classifying by the k closest training examples, and how the choice of k reshapes the resulting decision boundary — at a clear, approachable pace from the channel most consistently trusted for this kind of explanation.

Choosing k

A small value of k, such as one or three, bases each prediction on only a handful of the closest points, which follows local detail in the data closely but is highly sensitive to noise — a single mislabelled or unusual nearby point can swing the prediction. A large value of k averages over many more points, smoothing out that noise, but risks washing out genuinely local structure by including points that are not really representative of the area around the new one. The right value sits between these extremes and is typically chosen the same way any other hyperparameter is, covered in hyperparameter tuning: by trying several values and checking which performs best on data the model was not trained on.

Why scaling matters more here than almost anywhere else

K-nearest neighbors makes every prediction by computing distances, directly as covered in feature scaling and encoding, which makes it one of the models most sensitive to unscaled features of any covered in this roadmap. A feature measured in the tens of thousands will dominate a Euclidean distance calculation — see Euclidean norm for the underlying formula — over one measured between zero and one, regardless of which feature is actually more relevant to the prediction. Scaling every feature onto a comparable range before training is close to mandatory here, in a way it simply is not for a tree-based model.

The cost that shows up at prediction time, not training time

Most models do the expensive work during training and make predictions cheaply afterward. K-nearest neighbors inverts this: “training” is nearly free, since it amounts to storing the data, but every single prediction requires computing the new point’s distance to every stored training example. This means prediction cost grows directly with the size of the training set — a dataset with ten million training examples makes every single prediction noticeably more expensive than one with ten thousand, which is rarely true of a model like linear regression, whose prediction cost depends only on the number of coefficients, fixed once training finishes, and not on how large the training set originally was.