Choosing how many topics

Two different signals are shown here on the same 24-document cricket / curry / election corpus from earlier lessons: held-out perplexity for LDA, and reconstruction error for NMF. Neither hands you a clean answer — that is the point, and it is discussed honestly below.

Setup#

pip install scikit-learn

Outputs verified with scikit-learn 1.7.2 on CPU.

NMF's reconstruction error across several values of k#

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF

cricket = [
    "the batsman hit a six to win the match in the final over",
    "the bowler took three wickets as the batsman walked back after the match",
    "the team captain praised his batsman after the thrilling match",
    "the umpire signalled a boundary as the batsman completed the run",
    "the batsman and bowler shook hands after a hard fought match",
    "a six from the batsman sealed the match in the final over",
    "the bowler ran in fast and beat the batsman for a wicket",
    "the crowd cheered as the batsman brought up his century in the match",
]
curry = [
    "add chopped onion to hot oil then simmer the curry with salt",
    "the recipe needs turmeric, salt and slow simmering of the curry",
    "fry the spices in oil before you add the curry paste for dinner",
    "the chef seasoned the curry with salt and a spoon of turmeric",
    "simmer the curry slowly so the turmeric and salt blend into the oil",
    "the curry recipe calls for onion, turmeric and a pinch of salt",
    "heat the oil, add turmeric, then simmer the curry until thick",
    "the chef added salt and turmeric to the simmering curry pot",
]
election = [
    "the candidate promised new roads and hospitals before the election",
    "voters lined up outside the polling booth to vote in the election",
    "the election result was announced after votes were counted all night",
    "the losing candidate conceded the election after the final count",
    "the candidate campaigned for votes across the election constituency",
    "polling booths across the city saw voters queue for the election",
    "the winning candidate thanked voters after the election result",
    "election officials counted votes at the polling booth past midnight",
]
docs = cricket + curry + election

vectorizer = TfidfVectorizer(stop_words="english")
X = vectorizer.fit_transform(docs)

errors = []
for k in [2, 3, 4, 5, 6, 7, 8]:
    nmf = NMF(n_components=k, random_state=0, max_iter=500)
    nmf.fit(X)
    errors.append(nmf.reconstruction_err_)

prev = None
for k, err in zip([2, 3, 4, 5, 6, 7, 8], errors):
    drop = "" if prev is None else f"   drop from previous k: {prev - err:.2f}"
    print(f"k={k}  reconstruction error={err:.2f}{drop}")
    prev = err

k=2  reconstruction error=4.35
k=3  reconstruction error=4.11   drop from previous k: 0.25
k=4  reconstruction error=3.94   drop from previous k: 0.17
k=5  reconstruction error=3.79   drop from previous k: 0.15
k=6  reconstruction error=3.64   drop from previous k: 0.15
k=7  reconstruction error=3.50   drop from previous k: 0.14
k=8  reconstruction error=3.36   drop from previous k: 0.14

The part worth reading twice#

Reconstruction error goes down at every single value of k, without exception. Mathematically, it has no choice — more topics can only ever fit the data at least as well, never worse. There is no "bottom of the curve" to find here, because there isn't one.

Look at the drop column instead. It falls sharply from k=2 to k=3 (0.25), then settles into a much gentler, fairly steady decline (0.14–0.17) from k=3 onward. That flattening — not a dramatic dip, only the rate of improvement slowing down — is the "elbow" people mean when they say "look for the elbow." It is subtle here, and it is often this subtle on real data too. The true number of topics in this corpus is 3, and the elbow, while real, is not dramatic. Do not expect a sharp knee in the curve; expect exactly this kind of gentle levelling-off, and treat it as one signal among several, not a verdict.

The walkthrough#

Why NMF's error can never increase with more topics. Adding a topic gives the model one more building block to reconstruct the data with. In the worst case it ignores the extra block; it can never be forced to do worse. This is exactly why raw reconstruction error, alone, cannot select k — it will always vote for "more."

Perplexity has the identical problem for LDA, from the opposite direction. More topics generally lets LDA fit the training data better too. This is why any principled comparison must be done on a held-out split — data the model never trained on — using LatentDirichletAllocation.perplexity() on a test_counts matrix the model has not seen. Even then, as the researcher block covers, held-out likelihood and human-judged topic quality can point in different directions.

Reading the actual topics still matters more than either number. After picking a few candidate values of k using the drop-off pattern, print the top words for each topic at each k — exactly as in the LDA and NMF lessons — and judge for yourself whether the extra topics at higher k are genuinely distinct or only splitting one real theme in two.

Common mistakes#

Picking the k with the lowest error or perplexity, full stop. As shown above, that number keeps moving as k grows, on both metrics, for structural reasons that have nothing to do with topic quality.

Only ever trying one value of k. The entire technique here is comparative — a single reconstruction error number, on its own, tells you nothing. It is only informative next to the number one step below and one step above it.

Ignoring how the choice depends on what you'll do with the topics. A dashboard meant for a five-minute daily glance wants fewer, broader topics than a research project cataloguing every fine distinction in a corpus. The "right" k is partly a product decision, not only a statistical one.

Try it yourself#

Re-run the loop up to k=15 and watch whether the drop ever levels off completely to near zero, or keeps a small steady trickle going. On this tiny 24-document corpus, past k=8 you are well beyond the three real themes — the extra "topics" are mostly overfitting to individual documents.

What to learn next#

• Naming your topics automatically — once k is chosen, turning each topic's word list into a readable name.
• How many clusters? — the same "pick a count without a formula" problem, for clustering in general.
• Model evaluation — the broader idea of judging a model with more than one number.