Decision trees

Learning by asking questions

A decision tree predicts by asking a sequence of yes-or-no questions about the input, with each answer leading to a different branch, until a final answer is reached at a leaf. “Is income above fifty thousand? If yes, is age above forty? If yes, predict approved.” This is close to how a person might actually reason through a decision by hand, which is a large part of why trees are considered interpretable in a way few other models are — the entire chain of reasoning behind any single prediction can be read off directly, split by split.

How a split is actually chosen

At each node, the tree considers every feature and every possible threshold on that feature, and picks whichever single split makes the data on each side most homogeneous with respect to the outcome being predicted. Gini impurity is the standard way to measure that homogeneity for classification: a node where every example belongs to the same class has a Gini impurity of zero, and a node with an even mix of classes has a high one. The split chosen at each step is whichever one reduces this impurity the most across the two resulting branches, and the tree repeats this process recursively on each branch, splitting again and again until it reaches a stopping condition.

What to watch for

The video builds a classification tree from scratch, step by step, using Gini impurity exactly as described above, and also covers directly why a single tree tends to be unstable — both points named explicitly in this lesson’s scope. At just over eighteen minutes it runs a little long for a tight explainer, but it stays genuinely focused throughout rather than drifting.

Depth is the main overfitting control

A tree allowed to grow without limit will keep splitting until every leaf contains a single training example, perfectly memorising the training data in the process — and badly overfitting, since real-world new data will not match that exact memorised structure. Limiting the tree’s maximum depth, or requiring a minimum number of examples before a further split is allowed, are the standard ways to prevent this: both cap how finely the tree is permitted to carve up the training data, trading some training accuracy for a model that generalises better to new data, exactly the bias-variance tradeoff covered in generalization and overfitting.

Interpretable, and unstable

A tree’s full decision logic can be read directly, split by split, which makes it one of the most interpretable models in this roadmap — a real advantage whenever a stakeholder needs to understand not just what a model predicted but why. The cost is instability: because each split is chosen greedily based on what looks best at that one node, a small change in the training data — adding or removing a handful of examples — can change which split looks best near the top of the tree, and that change cascades into a substantially different tree below it, even when the underlying pattern in the data has barely shifted at all.

This specific weakness is exactly what motivates combining many trees together rather than relying on one, which is the whole idea behind bagging and random forests in the next group: average over many unstable trees, each trained on a slightly different view of the data, and the instability of any individual tree washes out in the average.