How gradient-boosted trees actually work

The first time I used XGBoost it felt like cheating. Import, fit, predict, and suddenly I'm near the top of the leaderboard. But "it just works" is a dangerous place to stop, because the day the model does something weird — overfits, ignores a feature you know matters, refuses to improve — you need a mental model of what those trees are actually doing. So this is the explanation I wish someone had handed me: how gradient-boosted trees work, built up from residuals, with no hand-waving.

Start with one dumb prediction

Forget trees for a second. Say you're predicting house prices. The simplest model in the world predicts the same number for every house — the mean of the training targets. It's wrong for almost everyone, but it's a starting point. Boosting's whole philosophy is: start with a weak guess, then add small corrections, one at a time.

Each correction is a decision tree. But here's the twist that trips people up — after the first guess, the trees are not trained to predict the price. They're trained to predict the residuals: how far off we currently are.

Trees that chase the errors

Suppose our constant guess was ₹50L and a particular house actually sold for ₹65L. The residual is +₹15L — we under-predicted. The next tree's job is to look at that house's features and learn "houses like this one need a nudge upward." We add a fraction of that nudge (scaled by the learning rate), recompute everyone's residuals, and grow another tree on the new residuals. Repeat a few hundred times. Each tree is shallow and weak; the ensemble is strong.

In real gradient boosting we don't literally use raw residuals — we use the gradient of the loss function. For squared error the gradient happens to be the residual, which is why the residual story is a perfectly good intuition. For other losses (log-loss for classification, for instance) the "error we chase" is just a different derivative. Same machine, different fuel.

What makes XGBoost different from plain boosting

Classic gradient boosting fits each tree to the gradient and calls it a day. XGBoost adds two things that matter enormously in practice: it uses the second derivative (the Hessian), and it bakes regularization directly into how splits are chosen. That second point is the real secret. Most explanations skip it, so let's not.

For every candidate leaf, XGBoost computes a similarity (or quality) score from the gradients and Hessians of the rows that land in it:

similarity = (sum of gradients)^2 / (sum of Hessians + lambda)

Here lambda is the L2 regularization term. Rows whose errors point the same direction pile up a big numerator and a high score; a leaf full of contradictory rows scores low. And lambda quietly shrinks every score toward zero, so noisy little leaves get discouraged.

Should I split this node? The gain calculation

When XGBoost considers splitting a node into a left and right child, it asks a single question: does splitting increase total similarity enough to be worth it? That's the gain:

gain = similarity(left) + similarity(right) - similarity(parent) - gamma

gamma is the minimum gain you demand before you're allowed to make a split — the "complexity tax." If the two children together aren't meaningfully purer than the parent, the gain is small, it doesn't clear gamma, and XGBoost refuses to split. This is pruning built into the growth rule itself, not bolted on afterward. When you tune gamma, lambda, and max_depth, you're literally moving the goalposts on this one inequality.

The learning rate is a humility knob

Once a tree is grown, XGBoost doesn't apply its full correction. It multiplies the output by a small learning rate (eta, often 0.01–0.1). Each tree only gets to nudge the prediction a little. It feels wasteful — why not take the whole step? — but small steps with many trees generalize far better than big steps with few. It's the difference between edging toward the answer and lunging past it. The trade-off is obvious: lower eta means you need more n_estimators, which is exactly where early stopping earns its keep.

import xgboost as xgb

model = xgb.XGBRegressor(
    n_estimators=3000,      # a high ceiling...
    learning_rate=0.03,     # ...because each step is small
    max_depth=5,            # weak learners, on purpose
    reg_lambda=1.0,         # L2 -> the similarity denominator
    gamma=0.1,              # min gain to split (pruning tax)
    subsample=0.8,          # row sampling per tree
    colsample_bytree=0.8,   # feature sampling per tree
)
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    early_stopping_rounds=100,   # stop when val loss stalls
)

Why the trees stay shallow

People coming from random forests expect deep trees. Boosting wants the opposite. Each tree is a weak learner — depth 3 to 6 is typical — because the ensemble's power comes from stacking many small corrections, not from any single tree being clever. A deep tree would fit the current residuals almost perfectly, leave nothing for the next tree to do, and memorize noise in the process. Shallow trees + a low learning rate + regularization is the trinity that keeps the whole thing from overfitting.

The mental model I actually carry around

• Prediction = a constant guess + a long sum of small tree corrections.
• Each tree is trained on the gradient of the loss (residuals, for MSE).
• Splits are chosen by gain; gamma and lambda decide when a split isn't worth it.
• The learning rate keeps every correction humble; early stopping picks the tree count for you.

Once I could see the residuals flowing from one tree to the next, tuning stopped feeling like superstition. A model that won't improve? Probably starved of gain — loosen gamma or add depth. A model that's memorizing the training set? Tighten the same knobs and drop the learning rate. The hyperparameters aren't magic numbers; they're just terms in an inequality the algorithm checks a few million times.

XGBoost isn't a black box. It's a very fast machine for asking "is this split worth it?" over and over — and the knobs you tune are just its definition of "worth it."