Bagging and random forests

The idea behind bagging

A single decision tree, as the previous lesson covered, is unstable — a small change to the training data can produce a noticeably different tree. Bagging, short for bootstrap aggregating, turns this instability into an advantage rather than fighting it directly: train many trees, each on its own random resample of the training data (drawn with replacement, so some examples appear more than once in a given resample and others are left out entirely), and combine their predictions by averaging for a continuous output or by majority vote for a classification. Each individual tree still overfits to the particular noise in its own resample, but that noise differs from tree to tree, while the real underlying signal, present throughout the original data, tends to agree across every resample. Averaging cancels the former while preserving the latter.

What to watch for

The video states directly what makes this work: bagging many trees and randomizing which features each one can split on reduces variance by decorrelating the trees from one another — exactly the mechanism this lesson needs to land, stated concisely rather than left implicit.

Random forests: bagging plus one more trick

A random forest is bagging applied specifically to decision trees, with one further addition. At every single split, rather than letting a tree consider every available feature, it is restricted to a random subset of them. This might seem like it should hurt each individual tree, and in isolation it does — but it serves a more important purpose across the ensemble as a whole: without this restriction, any one strong, obviously useful feature would dominate the top split of nearly every tree in the forest, making the trees highly similar to each other. Averaging over similar, correlated trees barely reduces variance at all, because they are all making roughly the same mistakes in roughly the same places. Forcing each tree to work with a different random subset of features at each split makes the trees genuinely differ from one another, which is precisely the condition under which averaging actually delivers a large reduction in variance.

Why decorrelation is the whole point

This is worth sitting with directly, because it is easy to average over many models and assume the benefit comes simply from “more models.” It does not — averaging over many highly correlated models, all making essentially the same errors, reduces variance only slightly. Averaging over many decorrelated models, each making different errors that are not all pointing the same direction, reduces variance substantially, because the errors genuinely cancel rather than reinforcing each other. This is the single most important idea in this lesson, and it is the reason random forests reliably outperform plain bagged trees by a meaningful margin in practice, despite the two methods looking almost identical on paper.

What a forest trades away

A random forest trades the single tree’s transparent, readable decision logic for better, more stable predictions — a forest of a few hundred trees has no simple chain of questions a person can read off the way a single tree does. This tradeoff, accuracy and stability against interpretability, recurs throughout this roadmap, and it is also exactly the trade that gradient boosting, the final lesson in this group, makes in a different way.