Gradient boosting

Correcting mistakes, one model at a time

Bagging, covered in the previous lesson, trains many models independently and averages them. Gradient boosting takes a fundamentally different approach: it builds models one at a time, in sequence, with each new model trained specifically to correct whatever the ensemble built so far still gets wrong. The first model makes an initial, usually quite crude prediction. The second model is trained not on the original targets, but on the residuals — the errors the first model made. The third model is trained on whatever errors remain after the first two are combined. This continues for as many rounds as specified, with the final prediction being the sum of every model’s contribution across the whole sequence.

What to watch for

The video builds up exactly this mechanism cleanly — starting from a naive first prediction, fitting successive trees to the residuals, and scaling each new tree’s contribution by a learning rate — which is the full core mechanism this lesson needs, from the channel most reliably clear on exactly this kind of step-by-step construction.

The learning rate: why not just fix it in one step

Each new tree in the sequence is capable of fitting the current residuals quite closely, and adding its full correction every time would let the ensemble fit the training data, including its noise, very aggressively and very fast. A learning rate — typically a small number like 0.1 — scales down each tree’s contribution before it is added to the ensemble, so progress toward fitting the data happens more gradually, over many more rounds. This costs more training iterations to reach a comparable fit, but the resulting ensemble is typically far more stable and generalises better, which is usually a trade well worth making.

Bias versus variance, boosting versus bagging

This lesson’s companion in the previous group reduces variance: many independent, parallel models, each a little different due to resampling, average out each other’s noise. Gradient boosting instead reduces bias: a single weak model, incapable on its own of capturing the full pattern in the data, is improved sequentially by each new model explicitly targeting what remains unexplained, building toward a far more accurate fit than any one of the individual trees could achieve alone. These are genuinely different mechanisms addressing genuinely different kinds of error, which is worth remembering rather than treating “ensemble method” as one undifferentiated category — the right choice between them depends on whether a single simple model is systematically missing the pattern (favouring boosting) or is unstable and noisy but roughly on target (favouring bagging).

XGBoost and LightGBM: gradient boosting, engineered

The specific algorithm described above is often implemented today through libraries like XGBoost and LightGBM, which add substantial engineering on top of the core idea — more efficient handling of missing values, built-in regularization to control overfitting directly, and considerably faster training on large datasets than a naive implementation would achieve. These names are worth recognising, because in practice “gradient boosting” very often means one of these specific, heavily optimised implementations rather than a from-scratch version of the algorithm, even though the underlying mechanism — sequential correction of residuals, scaled by a learning rate — is exactly what is described above in either case.