Preprocessing and Feature Selection
Boruta and shadow features
Boruta shuffles copies of your columns into meaningless "shadows", then keeps only the real columns that consistently beat the best shadow — a lie detector for feature importance.
- 8 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Boruta decides whether a column matters by making it compete against deliberately meaningless fakes of itself.
Suppose you suspect a "lucky charm" helps you win games. The honest test is a control: play some games with the real charm and some with a fake charm that looks identical. If the real one does not beat the fake, the luck was in your head.
Boruta builds that control experiment for every column. It copies each column and shuffles the copy, so the values are unchanged but their connection to the answers is destroyed. These scrambled copies are called shadow features — same ingredients, zero meaning.
Why it exists
Importance scores lie. Ask a random forest to rank columns and every column gets a positive score — even columns of pure random noise. The scores answer "how much did the model use this?", not "does this actually mean anything?". A model happily uses noise.
So the question "is this column's score high enough?" has no natural answer. High compared to what? Boruta's reply: compared to the best score achieved by a known fake. If a real column cannot outscore the luckiest shuffled column, its "importance" is indistinguishable from luck.
How it works
real columns shadow columns (shuffled copies)
age income clicks | sh_age sh_income sh_clicks
| |
+-------- train model on all of them --------+
|
importance scores for everything
|
did each real column beat the BEST shadow?
age: yes income: yes clicks: no
|
repeat many times, count the winsOne round proves nothing — a real column can lose once by chance. So Boruta repeats the tournament many times with fresh shuffles. Columns that beat the best shadow nearly always are confirmed. Columns that nearly never do are rejected. The undecided middle stays marked "unsure", and that honesty is a feature, not a flaw.
A real example you have seen
Medicine trials use the identical trick. Half the patients get the drug, half get a sugar pill that looks the same. The drug is approved only if it beats the sugar pill convincingly and repeatedly. Shadow features are sugar-pill columns.
Remember this
- Shadow features are shuffled copies: real values, destroyed meaning.
- A column earns its place by repeatedly beating the best shadow, not the average one.
- Every importance score is positive; only the comparison against fakes makes scores meaningful.
What to learn next
- Recursive feature elimination — squeezing the confirmed set down to a minimal one.
- Random forest — the engine Boruta runs on.
- SHAP and LIME — the importance scores modern Boruta variants use.
Developer — Code and libraries.
Setup
pip install scikit-learn numpyOutputs verified with scikit-learn 1.7.2. The original BorutaPy package exists but has lagged behind recent NumPy and sklearn releases; the loop below is the algorithm's core in twenty lines, dependency-free and easier to trust.
The tournament, by hand
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
X, y = make_classification(n_samples=400, n_features=8, n_informative=3,
n_redundant=1, random_state=0, shuffle=False)
rng = np.random.default_rng(0)
hits = np.zeros(X.shape[1], dtype=int)
trials = 20
for _ in range(trials):
shadows = rng.permuted(X, axis=0) # every column shuffled: signal destroyed
both = np.hstack([X, shadows])
forest = RandomForestClassifier(n_estimators=100, random_state=0).fit(both, y)
real = forest.feature_importances_[:X.shape[1]]
best_shadow = forest.feature_importances_[X.shape[1]:].max()
hits += (real > best_shadow) # beat the best fake? one hit
for i, h in enumerate(hits):
verdict = "keep" if h >= 15 else ("drop" if h <= 5 else "unsure")
print(f"feature {i}: beat shadows {h:2d}/{trials} -> {verdict}")feature 0: beat shadows 20/20 -> keep feature 1: beat shadows 20/20 -> keep feature 2: beat shadows 20/20 -> keep feature 3: beat shadows 20/20 -> keep feature 4: beat shadows 0/20 -> drop feature 5: beat shadows 0/20 -> drop feature 6: beat shadows 0/20 -> drop feature 7: beat shadows 1/20 -> drop
The walkthrough
The verdict matches the construction. The dataset has three informative columns plus one redundant blend of them (shuffle=False keeps all four first). All four beat every shadow in every trial. The four noise columns won at most once out of twenty — exactly the rate luck allows.
rng.permuted(X, axis=0) shuffles each column independently, up and down its rows. Each shadow keeps its exact distribution — same mean, same outliers — but its alignment with the rows, and therefore with y, is destroyed. That preservation is why shadows are a fair control: the forest cannot tell them apart from real columns by their values alone.
Why the best shadow? Eight shadows produce eight lucky scores each round, and comparing to the maximum asks: "did this real column beat everything luck produced?". Comparing to the average shadow would let mildly lucky noise columns through.
Feature 7 won once. One win in twenty is what chance delivers. This is why the loop repeats: any single round would occasionally crown a noise column.
Common mistakes
Too few trials. With five rounds, a real-but-weak column can lose the coin-flips and be rejected. Twenty is a floor; the published algorithm runs up to a hundred with a proper statistical test on the win counts.
Comparing against the mean shadow importance. This is the tempting shortcut and it quietly inflates your feature set. The maximum is the defence against multiple-comparison luck.
Shuffling rows of the whole matrix together. rng.permuted(X) without axis=0, or shuffling entire rows, preserves the row-to-label alignment or scrambles across columns — either way the control is broken. Each column must be shuffled on its own.
Treating "unsure" as "drop". Boruta's middle verdict means "not enough evidence either way". For a model going to production, keeping an unsure column is usually cheaper than losing a real signal. Rerun with more trials or more trees before deciding.
Forgetting what question Boruta answers. It finds all columns carrying real information — including redundant ones, like feature 3 above. It does not find the smallest set that predicts well. For a minimal set, follow with RFE on the survivors.
Try it yourself
Add a weak = y * 0.3 + rng.normal(0, 1, 400) column — real signal, badly drowned in noise. Rerun with 20 and then 100 trials and watch which verdict it lands on each time.
What to learn next
- Recursive feature elimination — squeezing the confirmed set down to a minimal one.
- Random forest — the engine Boruta runs on.
- SHAP and LIME — the importance scores modern Boruta variants use.
Researcher — Mathematics and papers.
The algorithm, formally
Kursa and Rudnicki (2010), Feature selection with the Boruta package, Journal of Statistical Software 36(11). Each iteration: extend X with column-wise permuted copies (shadows), fit a random forest, record Z-scores of importance (mean decrease in accuracy divided by its standard deviation across trees), and register a hit for feature j if Z_j exceeds MZSA — the maximum Z-score among shadow attributes. After T iterations, the hit count for an irrelevant feature follows Binomial(T, p) with p approximately the chance of beating the maximum of independent shadow scores under the null. A two-sided binomial test at each step confirms features whose hit counts are improbably high, rejects improbably low, and leaves the rest tentative; rejected features (and their shadows) are removed and the loop continues until all features are decided or a cap is reached.
All-relevant vs minimal-optimal
Boruta targets the all-relevant problem: find every feature whose removal changes the joint distribution P(Y | X) — including features made redundant by correlation. Minimal-optimal methods (lasso, RFE, mRMR) seek a smallest sufficient subset and will arbitrarily keep one of a correlated group. The distinction is decisive in scientific applications: for biomarker discovery, dropping a redundant-but-causal gene is a substantive error, while for a deployed model it is a free compression. Nilsson et al. (2007, JMLR) formalise both problems and show all-relevant selection is the harder one.
Bias inherited from the importance measure
Impurity-based (Gini) importances are biased toward high-cardinality and continuous features (Strobl et al., 2007, BMC Bioinformatics), and shadows inherit their column's cardinality — which partially, not fully, controls the bias. Permutation importance on out-of-bag samples is the measure the original Boruta assumes. Under strong feature correlation, permutation importance itself becomes unreliable (permuting one of a correlated pair creates unrealistic data points); conditional importance schemes (Strobl et al., 2008) address this at significant cost.
Cost and modern variants
Each iteration fits a forest on 2p features: O(T * trees * n log n * 2p) — heavy for wide data, which motivated BorutaShap (using SHAP values as the importance measure, with tree-SHAP's polynomial exactness) and gradient-boosting hosts replacing the forest. The shadow idea itself — augmenting with knockoff controls — reappears with stronger guarantees in Model-X knockoffs (Candes et al., 2018, JRSS B), which construct exchangeable fake features achieving finite-sample false discovery rate control, at the price of needing a model of the feature distribution.
What to learn next
- Recursive feature elimination — squeezing the confirmed set down to a minimal one.
- Random forest — the engine Boruta runs on.
- SHAP and LIME — the importance scores modern Boruta variants use.