XGBoost vs LightGBM: which gradient booster should you reach for?

Every time I start a tabular ML problem, the same fork in the road shows up: reach for XGBoost or LightGBM? Both are gradient-boosted decision tree libraries, both routinely top Kaggle leaderboards, and both will happily give you a strong baseline in ten lines of code. But they build their trees in fundamentally different ways, and that one difference drives almost every practical trade-off between them.

The one idea that separates them

Gradient boosting builds an ensemble of trees where each new tree corrects the errors of the ones before it. The question is how each tree grows.

• XGBoost grows level-wise (depth-wise). It expands every node on a level before moving deeper, producing balanced, symmetric trees. Growth is controlled mostly by max_depth.
• LightGBM grows leaf-wise (best-first). It splits the single leaf that reduces the loss the most, wherever it is in the tree. Trees come out deep and lopsided, and growth is controlled mostly by num_leaves.

Leaf-wise growth usually reaches a lower loss with fewer splits — which is a big part of why LightGBM is faster. It is also exactly why LightGBM overfits small datasets if you let num_leaves run wild. A useful sanity rule: keep num_leaves comfortably below 2^(max_depth).

Why LightGBM is usually the faster one

Two engineering tricks do most of the work:

• Histogram binning. Instead of evaluating every possible split point, LightGBM buckets continuous features into discrete bins (255 by default) and searches over bins. XGBoost also supports a histogram method (tree_method="hist"), which closed a lot of the original gap — but LightGBM was built around it from day one.
• GOSS & EFB. Gradient-based One-Side Sampling keeps the high-gradient (hard) rows and subsamples the easy ones; Exclusive Feature Bundling packs sparse, mutually-exclusive features together. On wide, sparse data this is a real speed and memory win.

Categorical features

This is the difference I feel most in day-to-day work. LightGBM handles categorical columns natively — you just pass categorical_feature and it does an optimized partitioning instead of forcing you into one-hot columns. With XGBoost I usually still reach for explicit encoding (target/ordinal) unless I opt into its newer experimental categorical support. Fewer one-hot columns means less memory and often a cleaner model.

# LightGBM: categoricals handled natively
import lightgbm as lgb

model = lgb.LGBMClassifier(
    num_leaves=31,          # the real capacity knob
    learning_rate=0.05,
    n_estimators=2000,
    subsample=0.8,
    colsample_bytree=0.8,
)
model.fit(
    X_train, y_train,
    categorical_feature=cat_cols,
    eval_set=[(X_val, y_val)],
    callbacks=[lgb.early_stopping(100)],
)

A side-by-side I actually keep in my head

How I actually choose

My decision rule after building a fair number of these models:

• Reach for LightGBM when the dataset is large, has many categorical features, or when training-loop speed matters (lots of CV folds, big hyperparameter searches). On the demand-forecasting project I built, LightGBM + Optuna with strict time-series CV was the combination that let me iterate fast enough to actually tune it — it landed at 89.4% accuracy.
• Reach for XGBoost when the dataset is smaller, when I want a hard-to-beat, well-understood baseline, or when I need the maturity of its ecosystem and documentation. Its level-wise trees are simply more forgiving when you haven't tuned yet.

Honestly? On a new problem I train both with sensible defaults and early stopping, look at the validation curves, and let the data settle the argument. They are close enough that the real wins come from feature engineering and honest cross-validation — not from the logo on the library.

The booster rarely loses you the competition. Leakage in your validation split does.