Learn AI Create the Future — with Pranay
Main site

Topic Modelling and Text Clustering

NMF for topics

NMF finds topics by breaking a document-word table into two smaller tables that are never allowed to go negative, avoiding a lot of the guesswork LDA needs.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

NMF finds topics by breaking a big table of word counts into two small tables of only positive numbers.

Think about mixing paint. Any wall colour you have ever seen came from combining a handful of base cans. Nobody ever adds "negative red." You cannot un-paint a wall. Every colour on that wall is some positive amount of each can, added together.

NMF, short for Non-negative Matrix Factorization, treats a pile of documents the same way. Every document's word pattern is rebuilt as a positive combination of a handful of "base topics." Never a subtraction, only addition. That single restriction — no negative amounts allowed — is the whole idea. It turns out to matter a lot.

Why it exists

LDA, from the previous lesson, is a full probability model. It has to imagine an entire story of how each word got picked. Finding the best guess involves randomness. Run it twice and you can get slightly different topics.

NMF skips the storytelling. It asks a plainer question instead: can this table become two smaller tables multiplied together, keeping every number positive? That is pure algebra, not probability. It usually settles into a more consistent answer, faster, on the same data.

How it works

document-word table  =  document-topic table   x   topic-word table
   (2,000 x 5,000)          (2,000 x 5)                (5 x 5,000)

  "how much of each      "how much of each        "how strongly each
   word in each doc"       topic is in each doc"    word belongs to each topic"

Every number in every table stays zero or positive.
No document ever contains "negative curry."

The two small tables are what you actually read afterward. One tells you each document's topic mixture. The other tells you each topic's defining words. These are the same two outputs LDA produces, reached by a different route.

A real example you have seen

Shopping sites sometimes group products into informal "styles." They notice that "floral," "cotton" and "summer" tend to co-occur in product descriptions. This kind of grouping often runs on NMF under the hood. Its positive-only output turns easily into a human-readable label: "this looks like the 'summer floral' group."

Remember this

  • NMF factors a document-word table into two smaller tables, with no negative numbers allowed anywhere.
  • It is algebra, not probability — no generative story, and (with a fixed starting point) no run-to-run randomness.
  • It tends to produce sharper, more distinctive topics than LDA on small or noisy corpora. The cost: it does not give you a proper probability of "how confident" a document's mixture is.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn

Outputs verified with scikit-learn 1.7.2 on CPU.

The exact same 24 documents LDA struggled with

nmf_demo.py
from collections import Counter
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF

cricket = [
    "the batsman hit a six to win the match in the final over",
    "the bowler took three wickets as the batsman walked back after the match",
    "the team captain praised his batsman after the thrilling match",
    "the umpire signalled a boundary as the batsman completed the run",
    "the batsman and bowler shook hands after a hard fought match",
    "a six from the batsman sealed the match in the final over",
    "the bowler ran in fast and beat the batsman for a wicket",
    "the crowd cheered as the batsman brought up his century in the match",
]
curry = [
    "add chopped onion to hot oil then simmer the curry with salt",
    "the recipe needs turmeric, salt and slow simmering of the curry",
    "fry the spices in oil before you add the curry paste for dinner",
    "the chef seasoned the curry with salt and a spoon of turmeric",
    "simmer the curry slowly so the turmeric and salt blend into the oil",
    "the curry recipe calls for onion, turmeric and a pinch of salt",
    "heat the oil, add turmeric, then simmer the curry until thick",
    "the chef added salt and turmeric to the simmering curry pot",
]
election = [
    "the candidate promised new roads and hospitals before the election",
    "voters lined up outside the polling booth to vote in the election",
    "the election result was announced after votes were counted all night",
    "the losing candidate conceded the election after the final count",
    "the candidate campaigned for votes across the election constituency",
    "polling booths across the city saw voters queue for the election",
    "the winning candidate thanked voters after the election result",
    "election officials counted votes at the polling booth past midnight",
]
docs = cricket + curry + election
true_label = ["cricket"] * 8 + ["curry"] * 8 + ["election"] * 8

vectorizer = TfidfVectorizer(stop_words="english")   # NMF prefers TF-IDF, not raw counts
X = vectorizer.fit_transform(docs)

nmf = NMF(n_components=3, random_state=0, max_iter=500)
doc_topics = nmf.fit_transform(X)

words = vectorizer.get_feature_names_out()
for i, topic in enumerate(nmf.components_):
    top = [words[j] for j in topic.argsort()[-6:][::-1]]
    print(f"topic {i}: {', '.join(top)}")

winners = doc_topics.argmax(axis=1)
topic_label = {}
for t in range(3):
    labels_here = [true_label[i] for i in range(len(docs)) if winners[i] == t]
    topic_label[t] = Counter(labels_here).most_common(1)[0][0]

right = sum(topic_label[t] == label for t, label in zip(winners, true_label))
print(f"\n{right}/{len(docs)} documents landed in the topic matching their true theme")
Output
topic 0: curry, turmeric, salt, oil, simmer, add
topic 1: batsman, match, final, bowler, sealed, win
topic 2: election, candidate, votes, voters, polling, result

24/24 documents landed in the topic matching their true theme

Same 24 sentences the LDA lesson got 18 out of 24 on. NMF, on this exact data, gets all 24. That gap is not an accident, and it is worth sitting with for a moment before moving on.

The walkthrough

TfidfVectorizer, not CountVectorizer. NMF is not modelling a word-drawing process, so there is no requirement for integer counts. TF-IDF's down-weighting of common words tends to sharpen NMF's topics further.

Why NMF wins here. LDA has to spread probability mass everywhere the Dirichlet prior nudges it, even when the evidence is thin, because it is fitting a full generative story on very little data. NMF minimises reconstruction error directly — with clean, well-separated vocabulary like this toy corpus, that direct objective has an easier time finding the factorisation a person would call correct on sight. This gap narrows sharply on real text, where vocabulary overlaps far more and NMF loses some of this advantage.

nmf.components_ rows are topics; doc_topics rows are document mixtures. Identical shape and meaning to the LDA output in the previous lesson — this is why the two algorithms are usually taught back to back and often swapped for each other in production code with only a few lines changed.

Common mistakes

Assuming NMF always beats LDA. It does not. NMF has no principled way to express "I'm not sure" the way a probability distribution does, and on messy real-world text with heavy vocabulary overlap, LDA's probabilistic smoothing sometimes generalises better. Try both on your own corpus.

Not scaling max_iter up. NMF's multiplicative-update solver can need hundreds of iterations to fully converge on more complex data. sklearn will warn you if it stops before converging — read that warning, do not silence it.

Forgetting NMF is non-convex. Two different random initialisations can land in two different local minima with different-looking topics. random_state pins this down for reproducibility, but does not guarantee you found the best possible factorisation.

Try it yourself

Switch the vectorizer back to CountVectorizer (matching what LDA used) and re-run. Notice the topics get noticeably fuzzier — TF-IDF weighting is doing real work here, not only cosmetic cleanup.

What to learn next

Researcher — Mathematics and papers.

The objective

Given a non-negative document-term matrix X of shape n x V (all entries >= 0), NMF seeks W (shape n x k) and H (shape k x V), both non-negative, minimising:

text
min over W, H >= 0 of  ||X - WH||_F^2

Where:

  • ||.||_F — the Frobenius norm, the square root of the sum of squared matrix entries.
  • W — the document-topic matrix (doc_topics in the code above).
  • H — the topic-word matrix (nmf.components_ above).

A Kullback-Leibler divergence variant, D_KL(X || WH), is also common (scikit-learn's beta_loss="kullback-leibler") and behaves more like a count-modelling loss, closer in spirit to LDA's assumptions.

Why non-convex, and why initialisation matters

Unlike PCA, this optimisation has no closed-form solution: fixing H and solving for W is a convex least-squares problem, and vice versa, but jointly optimising both is not convex overall. Lee & Seung's (2001) multiplicative update rules solve it by alternating:

text
H <- H * (W^T X) / (W^T W H)
W <- W * (X H^T) / (W H H^T)

Where * and / are elementwise multiplication and division. Each step is guaranteed not to increase the objective, but convergence is only to a stationary point, not a global optimum — hence sensitivity to the starting W_0, H_0.

NNDSVD (Boutsidis & Gallopoulos, 2008), scikit-learn's default initialisation (init="nndsvda"), seeds W and H from the SVD of X rather than random noise, giving faster, more consistent convergence than a random start — part of why NMF often looks more deterministic in practice than its non-convex objective would suggest.

Complexity

Each multiplicative-update iteration costs O(n*V*k), dominated by the matrix products above. For sparse X (the normal case for text), practical implementations exploit sparsity in the X-dependent terms, making a single iteration considerably cheaper than the dense bound suggests. Total cost scales with the number of iterations to convergence, which NNDSVD initialisation typically reduces relative to random starts.

Relation to other factorisations

NMF sits in the same family as PCA and truncated SVD — all three factor a matrix into a low-rank approximation. The non-negativity constraint is what makes NMF's factors directly interpretable as "topics" and "memberships": PCA components mix positive and negative loadings freely, which reads naturally as a direction in space but not as a human-nameable topic. This is precisely why NMF, not PCA, became the standard matrix-factorization choice for interpretable topic modelling.

Key references

  • Lee, D. & Seung, H. (1999). Learning the parts of objects by non-negative matrix factorization. Nature 401.
  • Lee, D. & Seung, H. (2001). Algorithms for Non-negative Matrix Factorization. NeurIPS.
  • Boutsidis, C. & Gallopoulos, E. (2008). SVD based initialization: A head start for nonnegative matrix factorization. Pattern Recognition 41(4).

Current state

NMF remains a standard baseline in topic-modelling toolkits for its speed and the readability of its output, and it generalises well beyond text — the same factorisation underpins parts-based decomposition in image analysis and gene-expression clustering in bioinformatics, wherever "positive parts add up to a whole" is a reasonable assumption.

What to learn next