Learn AI Create the Future — with Pranay
Main site

Clustering in Depth

How many clusters?

The data never announces its group count — the elbow method and the silhouette score are the two standard ways to make an honest choice.

Read these first

On this page 6
  1. Why this matters
  2. How it works
  3. A real example you have seen
  4. What is honestly hard here
  5. Remember this
  6. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Clustering algorithms need you to name the number of groups, and the data rarely settles it.

So we lean on scores that reward tidy, well-separated groups.

Think of sorting a big pile of family wedding photos into albums. Two albums feels too crude — the haldi and the reception get mixed. Forty albums is absurd — every album holds three photos. Somewhere in between, adding one more album stops making the collection feel tidier.

That "stops helping" moment is exactly what the standard tools look for.

Why this matters

Ask K-means for five groups and it will produce five, even if the data truly has three. It never complains. The algorithm answers the question you asked, not the question you should have asked.

So the burden is on you. The good news: two honest measurements do most of the work.

How it works

The first tool is the elbow method. Measure how tightly points hug their group centres, for every group count from two upward. Tightness always improves as you add groups. But it improves in a telling pattern:

tightness
  |  \
  |   \
  |    \
  |     \_
  |       \__ <- the elbow: gains fall off a cliff here
  |          \______
  |                 \_____
  +---|----|----|----|----|--
      2    3    4    5    6     number of groups

Before the elbow, each new group fixes a real problem. After it, each new group splits a genuine group in half for a tiny gain. Pick the count at the bend.

The second tool is the silhouette score. For each point it asks two things. How close are you to your own group? How close are you to the nearest other group? A point deep inside its own group scores near the top. A point sitting on a fence between groups scores near the bottom. Average this over everyone, and the group count with the highest average wins.

A real example you have seen

Streaming apps group viewers by taste to power their recommendation rows. Nobody knows the "true" number of taste groups in India — it does not exist. The teams pick a count where the groups stop getting tidier, exactly like the albums.

What is honestly hard here

Sometimes there is no elbow, and the silhouette scores are all mediocre. That result is information: the data may have no clean group structure at all. Forcing a number onto smooth, structureless data is a common and quiet mistake.

Remember this

  • The algorithm never checks whether your group count is sensible. You must.
  • Elbow: stop adding groups when the tidiness gain falls off a cliff.
  • Silhouette: prefer the count where points sit deep inside their own group.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn

Outputs verified with scikit-learn 1.7.2 on CPU. The blob generator is seeded, so these numbers should reproduce; last-digit drift across platforms is normal.

Both tools on one dataset

The data below is built with four true clumps, so we can catch the tools being right.

choose_k.py
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score

X, _ = make_blobs(n_samples=300, centers=4, cluster_std=1.2, random_state=42)

for k in range(2, 8):
    km = KMeans(n_clusters=k, n_init=10, random_state=0).fit(X)
    sil = silhouette_score(X, km.labels_)
    print(f"k={k}  inertia={km.inertia_:7.1f}  silhouette={sil:.3f}")
Output
k=2  inertia= 9666.8  silhouette=0.577
k=3  inertia= 2346.9  silhouette=0.735
k=4  inertia=  805.5  silhouette=0.752
k=5  inertia=  730.0  silhouette=0.633
k=6  inertia=  660.6  silhouette=0.530
k=7  inertia=  587.8  silhouette=0.472

The walkthrough

Read the inertia column for the elbow. From k=3 to k=4 inertia collapses from 2346.9 to 805.5. From k=4 to k=5 it barely moves: 805.5 to 730.0. The cliff is between 4 and 5, so the elbow says k=4.

Read the silhouette column for the peak. It climbs to 0.752 at k=4, then falls steadily. Both tools agree with the truth we planted.

Notice how close k=3 came. Silhouette 0.735 versus 0.752 is a narrow win. On real data the race is often this tight, and that is fine — it usually means both answers are defensible. Report the tightness, not only the winner.

n_init=10 matters here. A stuck run from a bad start would poison one row of this table and could move the apparent elbow. Initialisation and choosing k interact.

Common mistakes

Picking the k with the lowest inertia. Inertia falls forever as k grows; k equal to the number of points scores zero. The elbow is about the change in inertia, never its raw value.

Running silhouette on unscaled features. Silhouette is built from distances. One wide-ranged column decides everything, and the score becomes a measurement of that column alone. Scale first.

Treating silhouette as a universal judge. It rewards round, well-separated, similar-sized groups. Crescent or ribbon-shaped clusters score badly even when they are real. For such shapes, reach for HDBSCAN, which chooses the count itself.

Forgetting the business constraint. If the marketing team can run four campaigns, k=4 beats a statistically prettier k=11. A cluster count nobody can act on has no value.

Try it yourself

Change cluster_std to 3.0 so the four clumps bleed into each other, and rerun. Watch the elbow blur and the silhouette peak flatten and shift. That is what "the data does not support a clean k" looks like in numbers.

What to learn next

Researcher — Mathematics and papers.

Silhouette, formally

For point $i$, let $a(i)$ be its mean distance to members of its own cluster, and $b(i)$ the smallest mean distance to any other single cluster. Then:

$$ s(i) = \frac{b(i) - a(i)}{\max{a(i),\, b(i)}} $$

Where:

  • $a(i)$ — cohesion: average distance from $i$ to its cluster-mates.
  • $b(i)$ — separation: average distance to the best alternative cluster.
  • $s(i) \in [-1, 1]$ — negative values mean $i$ sits closer to another cluster than its own.

The score is Rousseeuw (1987), Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Computing all pairwise distances costs $O(n^2 d)$, which bites past roughly $10^5$ points; subsample or use silhouette_score(..., sample_size=...).

The gap statistic

Tibshirani, Walther and Hastie (2001) formalise the elbow. Compare $\log W_k$ (within-cluster dispersion at $k$) against its expectation under a null reference distribution — uniform draws in the data's bounding box or PCA-aligned box:

$$ \text{Gap}(k) = \mathbb{E}^*[\log W_k^{\text{ref}}] - \log W_k $$

Choose the smallest $k$ with $\text{Gap}(k) \ge \text{Gap}(k+1) - s_{k+1}$, where $s_{k+1}$ is the reference simulation's standard error. The reference distribution answers the question the raw elbow cannot: how much would inertia drop anyway, with no structure present?

Other criteria in live use

  • Calinski–Harabasz (1974): ratio of between- to within-cluster dispersion; cheap, $O(nd)$, in scikit-learn as calinski_harabasz_score.
  • Davies–Bouldin (1979): average worst-case cluster similarity; lower is better; also built in.
  • BIC/AIC under a mixture model: fit Gaussian mixtures across $k$ and penalise parameters — the likelihood framing K-means lacks.
  • Stability selection (Ben-Hur, Elisseeff, Guyon, 2002): cluster many subsamples; the right $k$ yields groupings that agree with each other under ARI-style comparison. Von Luxburg (2010) gives the caveats: stability also rewards degenerately coarse solutions.

The honest caveat

Hennig (2015), What are the true clusters?, argues the question is ill-posed without a stated purpose: different legitimate cluster concepts (density modes, compact balls, connected components) give different counts on the same data. Every index above encodes one concept. Choose the index that matches what a cluster means in your application, then let it choose $k$.

What to learn next