Comparing clusters to known labels
Cluster ids are arbitrary names, so accuracy is the wrong ruler — ARI and NMI measure whether two groupings agree, regardless of what the groups are called.
- 7 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Cluster numbers are made-up names, so matching them against true labels grades nothing.
What you must check is whether the groupings match.
Two teachers split the same class into project teams. Teacher one calls her teams Red, Blue, Green. Teacher two calls his 1, 2, 3. Every child who shared a team under teacher one also shares a team under teacher two. The two divisions are identical — the names differ, and names mean nothing.
Now grade teacher two by checking "did Red get called Red?" He scores zero. The grading is broken, not the teams.
Why this matters
This broken grading is one of the most common clustering mistakes in practice. A clusterer returns groups numbered 0, 1, 2. Your known labels — when you are lucky enough to have some — say "sports", "politics", "movies". Line the columns up, compute accuracy, and a perfect clustering can score anything at all, because the numbering is arbitrary. Rerun the same algorithm and the numbers may shuffle again.
The fix is a family of scores that ignore names entirely. They ask one question. Do pairs of items that sit together in one grouping also sit together in the other? Pairs have no names, so renaming cannot fool the score.
How it works
truth: {A, B, C} together | {D, E} together
clusters: {A, B, C} = "7" | {D, E} = "2"
pair check: A-B together in both? yes
A-C together in both? yes
C-D apart in both? yes -> perfect agreementTwo such scores cover nearly all real use. The adjusted Rand index (ARI) counts agreeing pairs. It then subtracts the agreement that pure luck would produce. Random grouping scores about zero; perfect grouping scores 1. The normalised mutual information (NMI) asks a different question. How much does knowing a point's cluster tell you about its true label? It is rescaled so 0 means nothing and 1 means everything.
A real example you have seen
News apps cluster articles about the same story. Editors keep a small set of hand-labelled stories to audit the clustering each week. The cluster ids change every single day — pair-based scores are the only way that audit can work at all.
Remember this
- Cluster ids are arbitrary names. Accuracy against labels is meaningless.
- ARI: pair agreement, corrected for luck. Zero means random, one means identical.
- NMI: how much the clusters tell you about the labels, on a 0–1 scale.
What to learn next
- Using clusters as features — putting a validated clustering to work downstream.
- How many clusters? — internal validation, when no labels exist at all.
- Model evaluation — the supervised evaluation toolbox these metrics extend.
Developer — Code and libraries.
Setup
pip install scikit-learnOutputs verified with scikit-learn 1.7.2.
The renaming trap, and the metrics that dodge it
from sklearn.metrics import accuracy_score, adjusted_rand_score, normalized_mutual_info_score
truth = [0, 0, 0, 1, 1, 1, 2, 2, 2]
renamed = [2, 2, 2, 0, 0, 0, 1, 1, 1] # identical grouping, different ids
messy = [0, 0, 1, 1, 1, 2, 2, 2, 0] # genuinely different grouping
print("renamed: accuracy =", round(accuracy_score(truth, renamed), 2))
print("renamed: ARI =", adjusted_rand_score(truth, renamed))
print("renamed: NMI =", normalized_mutual_info_score(truth, renamed))
print("messy: ARI =", round(adjusted_rand_score(truth, messy), 3))
print("messy: NMI =", round(normalized_mutual_info_score(truth, messy), 3))renamed: accuracy = 0.0 renamed: ARI = 1.0 renamed: NMI = 1.0 messy: ARI = 0.111 messy: NMI = 0.421
The walkthrough
Accuracy scored the perfect grouping at 0.0. Every item is in the "wrong" numbered cluster while every pair relationship is preserved. This single output line is the whole argument for never using accuracy on cluster labels.
ARI and NMI both give the perfect grouping 1.0, unmoved by the renaming. They only see structure.
On the genuinely different grouping they disagree in size — ARI 0.111, NMI 0.421 — and that is normal. ARI counts pair agreements; NMI measures shared information. ARI hovers near zero for near-random groupings; NMI stays more generous when partial structure exists. Pick one as primary and stay consistent across experiments.
When labels are only for auditing, three related scores add nuance: homogeneity_score (each cluster contains one label only), completeness_score (each label lands in one cluster only), and v_measure_score (their harmonic mean). A clustering that shatters one true group into five clusters is homogeneous but incomplete — the pair of numbers tells you which failure you have.
Common mistakes
Matching cluster ids to labels by hand, then computing accuracy. For equal counts of clusters and labels an optimal matching exists (the Hungarian algorithm), but with unequal counts the matching itself becomes a modelling choice. ARI and NMI need no matching at all.
Using unadjusted scores with many clusters. Plain mutual information and the plain Rand index both drift upward as the cluster count grows, rewarding pointless fragmentation. Use the adjusted versions — adjusted_rand_score, adjusted_mutual_info_score — whenever cluster counts differ between runs.
Grading on the same handful of labelled points you tuned on. The audit set stops auditing once you optimise against it — the same discipline as any train/test split.
Concluding the clustering is bad because agreement with labels is low. The clustering may have found real structure your labels do not describe — customer geography instead of customer taste. Low ARI against one labelling disproves nothing about usefulness.
Try it yourself
Build shattered = [0, 0, 1, 1, 2, 2, 3, 3, 4] against the same truth — each true group split in two. Predict which of homogeneity and completeness will be high and which low, then verify with homogeneity_score and completeness_score.
What to learn next
- Using clusters as features — putting a validated clustering to work downstream.
- How many clusters? — internal validation, when no labels exist at all.
- Model evaluation — the supervised evaluation toolbox these metrics extend.
Researcher — Mathematics and papers.
The Rand index and its correction
For groupings $U$ and $V$ over $n$ items, classify each of the $\binom{n}{2}$ pairs as together-in-both ($a$), apart-in-both ($b$), or mixed. The Rand index is $RI = (a + b) / \binom{n}{2}$ (Rand, 1971). Its flaw: the expected value under random labelling is not zero and depends on cluster sizes.
Hubert and Arabie (1985) correct it against the permutation model, in which both marginals are held fixed:
$$ ARI = \frac{RI - \mathbb{E}[RI]}{\max(RI) - \mathbb{E}[RI]} $$
Where $\mathbb{E}[RI]$ is the expectation over random pairings with the observed cluster sizes, computable in closed form from the contingency table. ARI is 1 for identical partitions, near 0 for random ones, and can go negative for adversarially disagreeing partitions. Cost is $O(n + rc)$ from the $r \times c$ contingency table.
Information-theoretic measures
With $P(u, v)$ the joint distribution from the contingency table, the mutual information is:
$$ I(U; V) = \sum_{u, v} P(u, v) \log \frac{P(u, v)}{P(u)P(v)} $$
Where $P(u)$ and $P(v)$ are the marginal cluster and label frequencies. NMI rescales by a mean of the entropies $H(U)$ and $H(V)$ — arithmetic in scikit-learn's default; geometric and min/max variants exist and change values, so state which you used.
Vinh, Epps and Bailey (2010), Information theoretic measures for clusterings comparison, is the definitive treatment. Two of its results matter in practice: plain MI and NMI carry a positive bias that grows with the number of clusters, and the adjusted mutual information (AMI) subtracts the expected MI under the permutation model, the exact analogue of ARI's correction.
Choosing among them
- Romano et al. (2016) recommend ARI when the reference partition has large equal-ish clusters, AMI when clusters are unbalanced or small.
- V-measure (Rosenberg and Hirschberg, 2007) decomposes into homogeneity and completeness, and equals NMI with arithmetic normalisation — useful because the two components diagnose how a clustering disagrees.
- Fowlkes–Mallows (1983) is the geometric mean of pairwise precision and recall; it degrades more gracefully under noise in one partition.
- All of the above assume flat, hard partitions. Overlapping or hierarchical structures need generalisations (omega index; hierarchy-aware B-cubed variants).
A final trap from Gösgens et al. (2021): pair-counting and information-theoretic families can rank the same set of candidate clusterings differently. Agreement metrics are themselves models of what "similar partitions" means.
What to learn next
- Using clusters as features — putting a validated clustering to work downstream.
- How many clusters? — internal validation, when no labels exist at all.
- Model evaluation — the supervised evaluation toolbox these metrics extend.