Handling missing data

Rubin's taxonomy#

Rubin (1976), Inference and Missing Data, Biometrika, defines the missingness mechanism through the distribution of the indicator matrix $R$, where $R_{ij}=1$ if $X_{ij}$ is observed. Partition data into observed $X_{\text{obs}}$ and missing $X_{\text{mis}}$.

MCAR — missing completely at random:

$$ p(R \mid X_{\text{obs}}, X_{\text{mis}}, \phi) = p(R \mid \phi) $$

MAR — missing at random:

$$ p(R \mid X_{\text{obs}}, X_{\text{mis}}, \phi) = p(R \mid X_{\text{obs}}, \phi) $$

MNAR — missing not at random: neither holds; $R$ depends on $X_{\text{mis}}$ even after conditioning on $X_{\text{obs}}$. Here $\phi$ denotes the parameters of the missingness process.

Under MAR, and when $\phi$ is distinct from the parameters of interest $\theta$, the mechanism is ignorable: likelihood-based inference on the observed data is valid without modelling $R$. Complete-case analysis is unbiased only under MCAR, which is strictly stronger.

The developer example is MNAR by construction, since the refusal probability is a function of income itself.

Why the missingness indicator works here#

Adding $R$ as a feature is often criticised in the statistics literature, and correctly so for inference: under MAR it can bias coefficient estimates, a point made by Jones (1996) and reiterated in Little and Rubin's textbook.

Prediction is a different objective. If the deployment distribution has the same missingness mechanism as training, then $R$ is a legitimately observable feature at prediction time, and conditioning on it is valid. The target becomes $p(Y \mid X_{\text{obs}}, R)$ rather than $p(Y \mid X)$, which is exactly the quantity you can act on.

Josse et al. (2019), On the consistency of supervised learning with missing values, formalise this: for prediction with the same mechanism at train and test time, imputation followed by a universally consistent learner is Bayes-consistent, and "impute then flag" is a natural way to make $R$ available. The two literatures disagree only because they optimise different objectives.

The assumption that fails loudly is a shift in the mechanism. Redesign the form so income becomes mandatory, and the flag column becomes constant, the learned coefficient becomes unreachable, and performance drops without any alert firing.

Multiple imputation#

Single imputation understates variance, because it treats an estimated value as observed. Rubin's multiple imputation draws $m$ completed datasets from the posterior predictive distribution of the missing values, analyses each, and combines by Rubin's rules:

$$ \bar{Q} = \frac{1}{m}\sum_{k=1}^{m} \hat{Q}_k, \qquad T = \bar{U} + \left(1 + \frac{1}{m}\right) B $$

Where $\hat{Q}_k$ is the estimate from dataset $k$, $\bar{U}$ the average within-imputation variance, and $B = \frac{1}{m-1}\sum_k (\hat{Q}_k - \bar{Q})^2$ the between-imputation variance. The $(1+1/m)$ factor corrects for finite $m$; $m$ between 5 and 20 is conventional.

MICE (van Buuren and Groothuis-Oudshoorn, 2011, mice: Multivariate Imputation by Chained Equations, JSS) implements this by cycling over columns, regressing each incomplete variable on the others. sklearn.impute.IterativeImputer is a single-imputation variant of the same idea; it does not give you Rubin's variance correction.

Multiple imputation matters for confidence intervals. For a point prediction served to a user, it usually does not repay its cost.

MNAR: what is actually available#

MNAR is not identifiable from observed data alone. Any estimate requires an untestable assumption. The two standard families:

Selection models factor $p(X, R) = p(X)\,p(R \mid X)$ and specify the second term parametrically — Heckman's bivariate normal formulation being the canonical case. Results are notoriously sensitive to the normality assumption.

Pattern-mixture models factor $p(X, R) = p(X \mid R)\,p(R)$, modelling the distribution separately per missingness pattern. The unidentified part is made explicit as a sensitivity parameter, which is a more honest presentation.

The practical recommendation across the literature is the same: perform a sensitivity analysis. Shift imputed values by $\delta$ across a plausible range and report how the conclusion changes. A conclusion that survives $\delta$ from $-1$ to $+1$ standard deviations is worth something; one that flips at $\delta = 0.2$ is not.

Native handling in gradient boosting#

XGBoost, LightGBM and CatBoost learn a default direction per split: missing values are routed left or right by whichever choice maximises the split gain. This is a learned, non-linear analogue of the missingness indicator, applied at every node rather than once globally.

Consequence: on tabular data with informative missingness, boosted trees often need no explicit imputation, and imputing first can lose information by destroying the pattern. When benchmarking imputation strategies, include "pass the NaNs straight to the booster" as a baseline. It wins more often than people expect. See XGBoost.

Reading#

• Rubin, Inference and Missing Data, Biometrika 1976.
• Little and Rubin, Statistical Analysis with Missing Data, 3rd edition, Wiley 2019.
• van Buuren and Groothuis-Oudshoorn, mice: Multivariate Imputation by Chained Equations in R, Journal of Statistical Software 2011.
• Josse et al., On the consistency of supervised learning with missing values, 2019 — arxiv.org/abs/1902.06931
• van Buuren, Flexible Imputation of Missing Data, 2nd edition — freely readable online, and the most practical treatment available.

What to learn next#

• Imbalanced data — the next way a table misleads you about itself.
• Feature engineering — turning a flag like this into a family of features.
• XGBoost — a model that handles blanks natively.