Explainability

Two families, and why the distinction is load-bearing#

Interpretable models are constrained so the model is the explanation: sparse linear models, short decision lists, generalised additive models, decision sets. The explanation is exact by construction.

Post-hoc explanations are separate models fitted to describe an unconstrained model. They are approximations with an error term that is rarely reported.

Rudin (2019), Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead, makes the strong argument: on structured tabular data with meaningful features, the accuracy cost of an interpretable model is frequently negligible, and a post-hoc explanation of a black box provides the appearance of accountability without the substance. The empirical claim — that the accuracy gap is small on tabular data — has held up reasonably well; the policy conclusion remains contested.

What "explanation" formally means#

There is no single definition, and the ambiguity causes most of the confusion in the literature. Three distinct targets:

Attribution. Assign each input feature a share of the output. Requires choosing a baseline $x'$ against which "contribution" is measured, and the choice materially changes the result. Additive attribution methods satisfy $f(x) - f(x') = \sum_i \phi_i$.

Counterfactual. Report the minimal change to $x$ that flips the decision. Wachter et al. (2017) solve

$$ \arg\min_{x'} \; \big(f(x') - y'\big)^2 + \lambda\, d(x, x') $$

where $x'$ is the counterfactual point, $y'$ the desired output, $d$ a distance capturing plausibility and actionability, and $\lambda$ the trade-off weight. This is the form most useful to an affected individual, because it is actionable, and the form most easily gamed, because it is actionable.

Mechanistic. Identify the internal computation. Circuit-level analysis of transformers — Elhage et al. (2021), A Mathematical Framework for Transformer Circuits; Olsson et al. (2022) on induction heads — reverse-engineers specific behaviours rather than attributing to input features. Far more expensive, and the only family that makes falsifiable claims about the mechanism.

Attribution methods are not reliable by default#

Several negative results deserve to be better known.

Adebayo et al. (2018), Sanity Checks for Saliency Maps, show that several popular saliency methods produce visually similar maps when model weights are randomised. A method that is insensitive to the model it purports to explain is not explaining that model. The randomisation test they propose is cheap and should be run before any saliency method is trusted.

Kindermans et al. (2017) show that attributions from several methods change under a constant shift of the input, despite the model's behaviour being unchanged — an unreasonable sensitivity to an arbitrary baseline.

Hooker et al. (2019), ROAR, benchmark attribution methods by removing the top-ranked features and retraining. Several widely used methods performed no better than a random ranking under this test.

Slack et al. (2020), Fooling LIME and SHAP, construct a classifier that behaves in a discriminatory way on the real data distribution while producing innocuous explanations, by detecting the off-manifold perturbations both methods use. Adversarial explanation is feasible, which matters if explanations become a compliance artefact.

Faithfulness versus plausibility#

The distinction (Jacovi and Goldberg, 2020) is the one to hold onto.

• Faithfulness — does the explanation reflect the model's actual computation?
• Plausibility — does it look convincing to a human?

Human-subject evaluation optimises plausibility. Attribution methods tuned on human ratings can become less faithful while scoring better. Faithfulness has to be measured mechanically: deletion and insertion curves, ROAR-style retraining, sufficiency and comprehensiveness on token subsets (DeYoung et al., 2020, ERASER).

Correlated features#

The permutation family evaluates $f$ at points off the data manifold when features are dependent. Hooker, Mentch and Zhou (2021) analyse the resulting extrapolation bias and recommend conditional variants. Conditional permutation preserves the joint distribution but changes the question being asked — from "does the model use this feature" to "does this feature carry information beyond the others". Both are legitimate; conflating them is not.

Papers#

• Rudin, Stop Explaining Black Box ML Models for High Stakes Decisions, Nature MI 2019 — arxiv.org/abs/1811.10154
• Adebayo et al., Sanity Checks for Saliency Maps, NeurIPS 2018 — arxiv.org/abs/1810.03292
• Hooker et al., A Benchmark for Interpretability Methods (ROAR), NeurIPS 2019 — arxiv.org/abs/1806.10758
• Slack et al., Fooling LIME and SHAP, AIES 2020 — arxiv.org/abs/1911.02508
• Wachter, Mittelstadt and Russell, Counterfactual Explanations without Opening the Black Box, 2017 — arxiv.org/abs/1711.00399
• Jacovi and Goldberg, Towards Faithfully Interpretable NLP Systems, ACL 2020 — arxiv.org/abs/2004.03685
• Strobl et al., Bias in random forest variable importance measures, BMC Bioinformatics 2007

What to learn next#

• SHAP and LIME — the two local methods you will actually be asked for.
• Model cards and documentation — publishing what the model uses.
• Decision trees — the model that explains itself for free.