Feature scaling and encoding

Why scale is not a cosmetic detail

Two features measured on wildly different scales — a person’s age, ranging roughly from zero to a hundred, alongside their annual income, ranging into the hundreds of thousands — can badly distort a model that relies on either distance or gradient-based optimisation. A distance calculation that sums differences across features will be dominated almost entirely by whichever feature happens to have the larger numeric range, regardless of which one actually carries more useful information about the problem. Bringing features onto a comparable scale before training is how this distortion is avoided.

Normalization rescales a feature into a fixed range, typically zero to one, by subtracting the minimum and dividing by the range. Standardization instead rescales so the feature has a mean of zero and a standard deviation of one, which handles outliers somewhat more gracefully since it does not anchor to the specific minimum and maximum values present in the data. Neither is universally correct; which to use depends on the specific model and on whether the data’s own distribution is close to normal.

What to watch for

The video isolates exactly the comparison this lesson needs — when and why normalization versus standardization matters, and specifically for which kinds of models — in a tight, focused explanation rather than burying the distinction inside a broader survey.

Why some models need this and others genuinely do not

Distance-based models — k-nearest neighbors is the clearest example, covered in its own lesson — compute distances directly between data points, so an unscaled feature with a large range will dominate every distance calculation regardless of its actual importance. Gradient-based models, including linear and logistic regression and every neural network, optimise by taking steps proportional to a gradient, and features on wildly different scales distort that gradient, typically making training slower and less stable. Both of these model families genuinely need scaling to perform well.

Tree-based models — decision trees, random forests, gradient boosting — are the clear exception. A tree splits a dataset on a threshold for one feature at a time, such as “is this value greater than 50”, and that decision is completely unaffected by whatever scale the feature happens to be measured on. Scaling a tree-based model’s input features does nothing useful at all, and it is a common beginner habit to scale everything reflexively regardless of which model is actually being used.

Encoding: making categories numeric

A model only ever operates on numbers, so a categorical variable — a colour, a city, a product type — has to be converted into numeric form before training, and the method chosen matters a great deal.

One-hot encoding creates a separate binary column for each possible category, with exactly one column set to one for any given row. This avoids implying any false ordering between categories, at the cost of adding many columns when a category has many possible values.

Ordinal or label encoding assigns each category a single integer. This is appropriate only when the categories have a genuine natural order — a rating of low, medium, high, say — and actively misleading otherwise, since it implies a false numeric distance between categories that have no inherent order at all, such as cities or colours.

Target encoding replaces each category with a number derived from the target variable itself — the average outcome observed for that category in the training data. This is powerful for categories with very many possible values, where one-hot encoding would create an unwieldy number of columns, but it carries a real risk of leaking information from the target into the features if done without care, which is why it is normally computed using only training folds during cross-validation rather than the whole dataset at once.