Naming your topics automatically

Setup#

pip install scikit-learn

Outputs verified with scikit-learn 1.7.2 on CPU.

Building class-based TF-IDF (c-TF-IDF) from scratch#

This is the exact mechanism BERTopic uses under the hood — implemented here in plain NumPy so nothing is hidden.

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer

cricket = [
    "the batsman hit a six to win the match in the final over",
    "the bowler took three wickets as the batsman walked back after the match",
    "the team captain praised his batsman after the thrilling match",
    "the umpire signalled a boundary as the batsman completed the run",
]
curry = [
    "add chopped onion to hot oil then simmer the curry with salt",
    "the recipe needs turmeric, salt and slow simmering of the curry",
    "fry the spices in oil before you add the curry paste for dinner",
    "the chef seasoned the curry with salt and a spoon of turmeric",
]
election = [
    "the candidate promised new roads and hospitals before the election",
    "voters lined up outside the polling booth to vote in the election",
    "the election result was announced after votes were counted all night",
    "the losing candidate conceded the election after the final count",
]
clusters = {"cluster_0": cricket, "cluster_1": curry, "cluster_2": election}

# collapse each cluster into one "mega-document" -- the whole trick behind c-TF-IDF
vectorizer = CountVectorizer(stop_words="english")
mega_docs = [" ".join(docs) for docs in clusters.values()]
counts = vectorizer.fit_transform(mega_docs).toarray()   # shape: (n_clusters, n_terms)
words = vectorizer.get_feature_names_out()

tf = counts / counts.sum(axis=1, keepdims=True)             # how common a word is INSIDE this cluster
avg_words_per_class = counts.sum(axis=1).mean()
term_freq_across_classes = counts.sum(axis=0)                # how common a word is OVERALL
idf = np.log(1 + avg_words_per_class / term_freq_across_classes)

c_tf_idf = tf * idf

for name, row in zip(clusters.keys(), c_tf_idf):
    top_idx = row.argsort()[-4:][::-1]
    label_words = [words[i] for i in top_idx]
    print(f"{name}: auto-label = '{'-'.join(label_words)}'")

cluster_0: auto-label = 'batsman-match-team-wickets'
cluster_1: auto-label = 'curry-salt-oil-turmeric'
cluster_2: auto-label = 'election-candidate-counted-conceded'

Three tidy labels, built from nothing but word counts and one formula — no human read a single document.

The walkthrough#

" ".join(docs) per cluster is the key move. Ordinary TF-IDF compares a word's importance across individual documents. c-TF-IDF first merges every document in a cluster into one giant "document," so the comparison happens across clusters instead. Everything downstream is standard TF-IDF maths applied to these merged texts.

tf measures local importance. A word's share of all words inside this one cluster's merged text — high for words that dominate the cluster's vocabulary.

idf measures how unusual that word is elsewhere. avg_words_per_class / term_freq_across_classes grows large when a word is rare across the other clusters, and the log(1 + ...) compresses that ratio into a sane range while keeping it always positive.

Multiplying tf * idf rewards words that are both common here and rare elsewhere — exactly the property a good label needs. A word that is common everywhere (like a leftover stop word) gets a small idf and drops out of the top-4 automatically, with no manual stop-word list required at this stage.

Common mistakes#

Labelling with raw word frequency instead of c-TF-IDF. The single most frequent word in a topic is very often a generic word that happens to be common in that domain overall ("said," "today," "app") rather than the word that actually distinguishes this topic from the others.

Trusting the auto-label without glancing at a few real documents from the cluster. "curry-salt-oil-turmeric" is a fair label here because the underlying documents are genuinely about cooking. On messier real data, a label can look plausible and still miss the point of the cluster — a quick sanity read of two or three source documents catches this cheaply.

Using too few or too many words in the label. Two words is often ambiguous; eight words stops reading like a label and starts reading like the same ranked list you were trying to summarise. Three to five words is a reasonable working default.

Try it yourself#

Add a fourth, deliberately overlapping cluster — reuse a few cricket sentences mixed with a few curry sentences — and recompute. Watch the idf term pull previously "distinctive" words like curry down, since it no longer belongs to only one cluster.

What to learn next#

• Topic modelling on tweets and short text — labelling gets harder when there is barely any text per document to draw words from.
• BERTopic — the full pipeline this labelling step plugs into.
• TF-IDF — the single-document version of the same underlying idea.