Ensembles and Gradient Boosting
LightGBM
LightGBM makes gradient boosting fast by sorting feature values into coarse bins and growing trees leaf by leaf, which is why it dominates on large tables.
- 8 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
LightGBM is a gradient boosting library that trades tiny amounts of precision for enormous speed, using two tricks: binning and leaf-wise growth.
Think of a shopkeeper counting the day's coins. Weighing every coin individually would take all night. Instead he uses a sorting tray: coins fall into slots by size, and he counts slots. He loses nothing that matters and finishes in minutes.
LightGBM does this to your data. It does not consider every exact value of a feature when choosing a split. Instead it sorts values into about 255 bins, which are coarse slots. Only the slot boundaries get tested. That single decision makes training dramatically faster on big tables.
Why this had to be invented
Gradient boosting was winning every tabular competition by the mid-2010s, but training crawled once tables reached millions of rows. Finding the best split point meant examining huge numbers of candidate values, feature by feature, tree after tree.
Microsoft Research released LightGBM in 2017 — "light" as in lightweight — to break that bottleneck. The binning trick shrinks the work per split. The second trick changes which splits happen at all.
How it works
Most libraries grow trees level by level: split every branch, then every branch of every branch. LightGBM grows leaf-wise: at each step, find the one leaf whose split would reduce error most, and split only that leaf.
level-wise (the classic way) leaf-wise (LightGBM)
o o
/ \ / \
o o ← split both o o
/\ /\ /\
o o o o o o ← only the most
/\ profitable leaf
o o splits againFixing the leakiest hole in the roof first, instead of patching every room a little. The result is deeper, lopsided trees that reduce error faster per split — and overfit faster too, if you let them.
A real example you have seen
LightGBM came out of Microsoft's need to rank web search results. It then spread everywhere tables live. Fraud checks when you tap pay, delivery-time estimates on food apps, credit scoring, click prediction. For roughly a decade of Kaggle tabular competitions, the winner's stack has usually contained LightGBM, XGBoost, or CatBoost.
Remember this
- LightGBM = gradient boosting + binned features + leaf-wise growth.
- Built for large tables — that is where its speed advantage shows.
- Leaf-wise trees are powerful and quick to overfit; the leash is
num_leavesand early stopping.
What to learn next
- CatBoost — the sibling library built around categorical columns.
- Tuning gradient-boosted trees — which of these knobs matter, in order.
- XGBoost — the library that started the era, and the mechanics both share.
Developer — Code and libraries.
Setup
pip install lightgbmThe wheel is small — about 1.4 MB — with no heavy dependencies beyond numpy and scipy. Outputs verified with lightgbm 4.7.0 and scikit-learn 1.7.2 on CPU; the whole script runs in a few seconds.
Version note: lightgbm 4.7 renamed the validation-data arguments. The old fit(eval_set=[(X, y)]) pattern, which most tutorials still show, now emits a deprecation warning. The current spelling is eval_X= and eval_y=.
Train with early stopping
Ask for far more trees than needed, and let the validation set decide when to stop.
import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=5000, n_features=20, n_informative=8,
flip_y=0.05, random_state=5)
Xtr, Xval, ytr, yval = train_test_split(X, y, test_size=0.3, random_state=5)
model = lgb.LGBMClassifier(n_estimators=2000, learning_rate=0.05,
num_leaves=31, random_state=5, verbose=-1)
model.fit(Xtr, ytr, eval_X=Xval, eval_y=yval,
callbacks=[lgb.early_stopping(50, verbose=False)])
print("trees asked for:", 2000)
print("trees actually kept:", model.best_iteration_)
print("validation accuracy:", round(model.score(Xval, yval), 3))trees asked for: 2000 trees actually kept: 109 validation accuracy: 0.909
We requested 2,000 trees; validation loss stopped improving at 109, the callback waited 50 more rounds to be sure, and training ended there. Predictions automatically use the best iteration.
The walkthrough
num_leaves=31 is the main capacity knob, not max_depth. A leaf-wise tree with 31 leaves can be deep and lopsided. Doubling num_leaves roughly doubles what each tree can memorise — it is the first thing to tune, as covered in tuning gradient-boosted trees.
early_stopping(50) watches the validation metric and stops after 50 rounds without improvement. This is how n_estimators should always be set in practice: too high on purpose, then cut by the data.
verbose=-1 silences per-iteration logging. Without it, real runs bury your terminal.
Where the speed shows. On this toy table you will not feel the difference. On a few million rows, LightGBM commonly trains several times faster than exact-split methods — that, not accuracy, is its core advantage. Accuracy across LightGBM, XGBoost and CatBoost is usually within noise of each other once tuned.
Common mistakes
No early stopping. A fixed n_estimators=100 is either wasted budget or an overfit — you cannot know which without a validation set. Always pair a large n_estimators with the callback.
Cranking num_leaves while leaving max_depth unset. num_leaves=1024 builds trees that can memorise 5,000 rows outright. If validation accuracy falls while training accuracy climbs, shrink num_leaves first. The pattern is ordinary overfitting wearing new clothes.
Using LightGBM on tiny datasets. Below a few thousand rows, its defaults (min_child_samples=20, binning) are tuned for scale, and simpler models with cross-validation are usually steadier. LightGBM's advantages need data to show.
Copying eval_set from older tutorials. Works today with a LGBMDeprecationWarning; will break eventually. Use eval_X / eval_y with lightgbm ≥ 4.7.
Try it yourself
Set num_leaves to 8, 31 and 256, rerunning each time. Record kept trees and validation accuracy. Notice the compensation: smaller leaves → more trees kept. Then break the early stopping on purpose — remove the callback and set n_estimators=2000 — and compare validation accuracy against the stopped run.
What to learn next
- CatBoost — the sibling library built around categorical columns.
- Tuning gradient-boosted trees — which of these knobs matter, in order.
- XGBoost — the library that started the era, and the mechanics both share.
Researcher — Mathematics and papers.
The histogram algorithm
Exact greedy split finding scans sorted feature values: $O(n)$ per feature per node after an $O(n \log n)$ presort, with cache-hostile access patterns. LightGBM discretises each feature into $B$ bins (default 255) once, up front. Split finding then builds a histogram of gradient statistics per bin:
- Histogram construction: $O(n d)$ per level of tree depth, for $n$ rows and $d$ features.
- Split evaluation: $O(B d)$ per node — independent of $n$.
- The histogram subtraction trick: a node's sibling histogram equals parent minus node, halving construction work.
Reference: Ke et al. (2017), LightGBM: A highly efficient gradient boosting decision tree, NeurIPS. The binning idea itself is older (McRank, and Fayyad and Irani's discretisation); LightGBM's contribution was making the whole pipeline histogram-native, plus two sampling schemes:
GOSS (gradient-based one-side sampling) — keep the top $a$ fraction of rows by gradient magnitude, sample a fraction $b$ of the rest, and reweight the sampled small-gradient rows by $\frac{1-a}{b}$ to keep the gradient distribution unbiased. Rows the model already fits well contribute little signal and are mostly dropped.
EFB (exclusive feature bundling) — mutually exclusive sparse features (never simultaneously non-zero, common after one-hot encoding) are merged into single features, reducing effective $d$.
Leaf-wise growth
Level-wise growth splits all leaves at a depth; leaf-wise (best-first) growth repeatedly splits $\arg\max_{\ell} \Delta L_\ell$, the leaf with the largest loss reduction. For a fixed leaf budget, best-first trees achieve lower training loss — they allocate capacity where the loss surface demands it — at the price of higher variance. The regularisers are num_leaves, max_depth, min_child_samples (minimum rows per leaf) and min_split_gain.
The gain formula per split is standard second-order gradient boosting, shared with XGBoost (Chen and Guestrin, 2016, XGBoost: A scalable tree boosting system, KDD):
$$ \Delta L = \frac{1}{2}\left[ \frac{G_L^2}{H_L + \lambda} + \frac{G_R^2}{H_R + \lambda} - \frac{(G_L + G_R)^2}{H_L + H_R + \lambda} \right] - \gamma $$
Where $G$ and $H$ are sums of first and second derivatives of the loss over the rows falling left ($L$) and right ($R$), $\lambda$ is L2 leaf regularisation, and $\gamma$ is the fixed cost of adding a leaf.
Categorical features
LightGBM accepts integer-coded categoricals directly and finds splits by sorting categories by their gradient statistics — the Fisher (1958) optimal-partition trick — rather than one-hot encoding. This is effective but leak-prone at high cardinality; CatBoost exists largely because of that leak, and the next lesson dissects it.
Ecosystem position
The histogram approach won so decisively that its competitors adopted it: XGBoost's tree_method="hist" and scikit-learn's HistGradientBoosting* estimators are the same design. Comparative studies (Bentéjac, Csörgő and Martínez-Muñoz, 2021, A comparative analysis of gradient boosting algorithms, Artificial Intelligence Review) find the three major libraries statistically close in accuracy, with LightGBM typically fastest to train. On tabular benchmarks against deep learning, boosted trees remain the reference point (Grinsztajn et al., 2022).
What to learn next
- CatBoost — the sibling library built around categorical columns.
- Tuning gradient-boosted trees — which of these knobs matter, in order.
- XGBoost — the library that started the era, and the mechanics both share.