Collecting data

Selection as conditioning on a collider#

Let $S \in {0,1}$ indicate whether a unit enters the sample. Your training distribution is $p(x, y \mid S = 1)$; the deployment distribution is $p(x, y)$. The two coincide only when $S \perp (X, Y)$.

Three regimes, in increasing severity:

Covariate shift. $p_{\text{train}}(x) \neq p_{\text{test}}(x)$ but $p(y \mid x)$ is shared. Selection depends on $X$ alone: $S \perp Y \mid X$. This is correctable in principle by importance weighting with $w(x) = p_{\text{test}}(x) / p_{\text{train}}(x)$, provided the support condition $p_{\text{train}}(x) > 0$ wherever $p_{\text{test}}(x) > 0$ holds.

Label shift. $p(y)$ changes but $p(x \mid y)$ is shared. Correctable via the confusion-matrix estimator of Saerens et al. (2002) or BBSE (Lipton et al., 2018, Detecting and Correcting for Label Shift with Black Box Predictors).

Concept shift. $p(y \mid x)$ itself differs across strata. This is the developer example: the rural and urban conditionals are different functions. Reweighting cannot fix it, because the target function is not shared. The only remedies are collecting the missing stratum or fitting per-stratum models.

Distinguishing these three before choosing a correction is the whole game, and it is frequently skipped.

The support condition is the binding constraint#

Importance weighting is often quoted as the general fix for biased sampling. Its variance is governed by:

$$ \operatorname{Var}!\left[\hat{R}w\right] \;\propto\; \mathbb{E}{p_{\text{train}}}!\left[ w(X)^2 \ell(X)^2 \right] $$

Where $w(X)$ is the density ratio and $\ell$ the loss. As $p_{\text{train}}(x) \to 0$ in a region where $p_{\text{test}}(x)$ is positive, $w$ diverges and the estimator's variance explodes. The effective sample size, $\left(\sum_i w_i\right)^2 / \sum_i w_i^2$, collapses.

In the developer example with an 8% rural sample, weights would exceed 20 on the sparse stratum. The reweighted estimate would be nominally unbiased and practically worthless. Reweighting cannot manufacture coverage.

Heckman correction, and why it is rarely used here#

Heckman (1979), Sample Selection Bias as a Specification Error, Econometrica, models selection and outcome jointly, estimating a selection equation and adding the inverse Mills ratio as a regressor in the outcome equation. Consistency requires a valid exclusion restriction: a variable that drives selection but not the outcome.

That is a strong requirement. In ML settings such an instrument is seldom available and seldom defensible, which is why the technique appears far more often in econometrics than in applied machine learning.

Sampling designs worth knowing#

• Stratified sampling. Partition by a known variable, sample within strata. Variance of a stratum-weighted mean estimator is $\sum_h W_h^2 \sigma_h^2 / n_h$, minimised by Neyman allocation $n_h \propto W_h \sigma_h$. When strata differ in conditional structure, oversampling small strata is the correct move even though it distorts marginals — reweight at evaluation time, not collection time.
• Cluster sampling. Cheaper in the field, but the design effect $\text{deff} = 1 + (m-1)\rho$ inflates variance, where $m$ is cluster size and $\rho$ the intra-cluster correlation. Ten villages of 100 people is nowhere near 1,000 independent observations.
• Active learning. Uncertainty sampling and query-by-committee (Settles, 2009, Active Learning Literature Survey) reduce labelling cost, but the resulting pool is deliberately non-representative. Keep a separately drawn random slice for unbiased evaluation.

Documentation as a first-class artefact#

• Gebru et al. (2021), Datasheets for Datasets, CACM. A standard questionnaire covering motivation, composition, collection process, preprocessing, uses and distribution.
• Bender and Friedman (2018), Data Statements for NLP, TACL. Focused on speaker demographics, curation rationale and dialect, which are exactly the axes that go undocumented.
• Mitchell et al. (2019), Model Cards for Model Reporting, FAccT. Requires disaggregated evaluation across groups — the practice the developer section demonstrates.

Empirical work on dataset-level bias#

Torralba and Efros (2011), Unbiased Look at Dataset Bias, CVPR, ran a "name that dataset" experiment: a classifier trained to identify which corpus an image came from reached far above chance. They also measured cross-dataset generalisation, finding consistent performance drops when training on one corpus and testing on another.

Buolamwini and Gebru (2018), Gender Shades, FAccT, evaluated commercial gender classifiers disaggregated by skin type and gender, reporting error rates up to 34.7% for darker-skinned women against 0.8% for lighter-skinned men. The aggregate accuracy of those systems looked strong.

Both results share a structure: aggregate metrics concealed the failure, and disaggregated evaluation exposed it.

Practical protocol#

1. Write the target population down before collecting anything.
2. Record the selection mechanism, including the channel and any incentive offered.
3. Draw a small probability sample separately, at whatever cost, and reserve it purely for evaluation.
4. Report every metric disaggregated by the strata you can observe.
5. Track the effective sample size per stratum, not only the total row count.

Reading#

• Heckman, Sample Selection Bias as a Specification Error, Econometrica 1979.
• Torralba and Efros, Unbiased Look at Dataset Bias, CVPR 2011.
• Buolamwini and Gebru, Gender Shades, FAccT 2018.
• Gebru et al., Datasheets for Datasets, CACM 2021 — arxiv.org/abs/1803.09010
• Lipton, Wang and Smola, Detecting and Correcting for Label Shift with Black Box Predictors, ICML 2018 — arxiv.org/abs/1802.03916

What to learn next#

• Labelling data — turning collected rows into trustworthy answers.
• Bias in datasets — what a skewed sample does to real people.
• Train, test and validation splits — building a test set that resembles the world.