Learn AI Create the Future — with Pranay
Main site

Dimensionality Reduction

Truncated SVD and LSA

Truncated SVD compresses huge sparse matrices like word counts without destroying their sparsity, and applied to text it becomes LSA — grouping documents by theme rather than exact words.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Truncated SVD compresses a giant table into a few underlying themes — and on text, those themes group documents that never share a single word.

Ask a librarian what their ten thousand books are about, and they will not recite ten thousand titles. They will say: "mostly history, some science, a shelf of poetry." Thousands of books, three themes. Each book is then some blend — a history of science sits between two shelves.

Truncated SVD is the librarian for tables. Give it a table of which document uses which word, thousands of columns wide. It discovers the few themes underneath. It also gives each document's blend of those themes.

Why it exists

Word-count tables have two brutal properties. They are huge — one column per distinct word, easily 50,000. And they are sparse, meaning almost every cell is zero: a 200-word review uses 200 of the 50,000 columns, so 99.6% of the table is zeros. Computers store sparse tables cleverly, keeping only the non-zeros.

PCA would be the natural compressor, but PCA begins by centring — subtracting each column's average. Subtract anything from a sea of zeros and the zeros become non-zeros: the clever storage explodes into memory-eating dead weight. Truncated SVD is the variant that skips centring, so sparse stays sparse. That single technical difference is why it exists as a separate tool.

Applied to word tables, it earns a second name: latent semantic analysis (LSA) — "finding the hidden meaning". Documents about "wicket" and "stadium" get pulled near documents about "bat" and "ball", because those words keep appearing together across the collection. The themes emerge from co-occurrence, with nobody defining them.

How it works

            50,000 word columns                    2 theme columns
document 1 [0 0 1 0 ... 2 0 0 1]                  [0.83  0.00]  <- cricket-y
document 2 [1 0 0 0 ... 0 3 0 0]    ────────>     [0.71  0.00]  <- cricket-y
document 3 [0 2 0 1 ... 0 0 0 0]    truncated     [0.00  0.78]  <- finance-y
                                    SVD

"Truncated" means you keep only the top few themes and cut the rest — the compression is the truncation.

A real example you have seen

Search engines learned this trick early. Search "how to fix a puncture" and you get pages about "repairing a flat tyre" — no shared words. The engine knows "puncture" and "flat tyre" belong to the same theme because millions of pages use them in the same company.

Remember this

  • Truncated SVD is PCA's cousin that keeps sparse data sparse.
  • On text it is called LSA: documents cluster by theme, not by exact words.
  • Each document becomes a small vector of theme strengths.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn

Outputs verified with scikit-learn 1.7.2.

Six headlines, two themes

tsvd.py
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD

docs = [
    "cricket bat ball wicket stadium",
    "wicket fell to a great ball at the stadium",
    "bat swings, ball flies, crowd roars",
    "rupee falls against dollar, markets slide",
    "stock markets rally as rupee gains",
    "dollar strong, stock prices slide",
]

tfidf = TfidfVectorizer()
X = tfidf.fit_transform(docs)                  # sparse: 6 docs x vocabulary
print("tf-idf shape:", X.shape)

svd = TruncatedSVD(n_components=2, random_state=0)
Z = svd.fit_transform(X)                       # works directly on sparse input

words = tfidf.get_feature_names_out()
for i, comp in enumerate(svd.components_):
    top = comp.argsort()[-4:][::-1]
    print(f"direction {i}: {', '.join(words[t] for t in top)}")
print("doc positions:")
print(Z.round(2))
Output
tf-idf shape: (6, 26)
direction 0: ball, wicket, stadium, bat
direction 1: dollar, slide, rupee, markets
doc positions:
[[ 0.83 -0.  ]
 [ 0.71  0.  ]
 [ 0.56 -0.  ]
 [ 0.    0.78]
 [ 0.    0.64]
 [ 0.    0.68]]

The tiny numbers can differ in the last decimals across sklearn and SciPy versions — the split into two clean themes is the stable part.

The walkthrough

Nobody told it about cricket. Direction 0's top words are ball, wicket, stadium, bat; direction 1's are dollar, slide, rupee, markets. The themes fell out of which words co-occur. The first three documents score high on direction 0 and about zero on direction 1; the finance headlines mirror that. Documents 1 and 2 share the theme despite different word choices — that is the LSA effect.

tf-idf before SVD is the standard pairing. Raw counts let common words dominate the variance; tf-idf downweights words that appear everywhere, so the directions align with distinguishing vocabulary. See text classification for tf-idf itself.

Negative zeros and arbitrary signs. That -0. is not a bug: each direction's overall sign is arbitrary — flipping every weight in a direction changes nothing. Never interpret the sign of a whole component; interpret contrasts within it.

random_state=0 matters because sklearn's solver (algorithm="randomized") uses randomised linear algebra for speed on wide matrices. Fix the seed for reproducible components; use algorithm="arpack" for the deterministic-but-slower path.

Common mistakes

Densifying to use PCA. PCA().fit(X.toarray()) on a 100,000-document tf-idf matrix asks for hundreds of gigabytes. If your matrix is sparse, TruncatedSVD is not the alternative — it is the method.

Reading every component as a clean topic. The first few directions are often interpretable; deeper ones become "everything left over" mixtures with both large positive and negative weights. If you need parts that read as topics, NMF is built for that.

Skipping normalisation before cosine similarity. SVD outputs have wildly different lengths per document. For semantic search over the reduced vectors, L2-normalise them (sklearn.preprocessing.Normalizer) — the classic "LSA pipeline" is tf-idf, SVD, normalise. This is also the ancestor of modern embeddings.

Keeping too few components for retrieval. Two components make a demo; real LSA for search historically used 100 to 300. Below that, unrelated themes get squeezed together.

Try it yourself

Add a seventh headline mixing both themes — "stadium owners watch ticket markets slide" — and print its position. Predict before running: which directions should light up?

What to learn next

Researcher — Mathematics and papers.

The decomposition

Any matrix X (n x d) factors as X = U Sigma V^T with U, V orthonormal and Sigma diagonal with singular values sigma_1 >= sigma_2 >= ... >= 0. Truncation keeps the top k: X_k = U_k Sigma_k V_k^T. The Eckart-Young theorem (1936) makes this optimal: X_k minimises ||X - B|| over all rank-k matrices B, in both Frobenius and spectral norm — truncated SVD is the best possible rank-k linear compression, full stop. Squared Frobenius error equals the sum of discarded sigma_i^2.

Relation to PCA: PCA is SVD applied to the centred matrix. Uncentred SVD's first component tends to point at the data's mean direction; on tf-idf matrices, where all entries are non-negative, that first component is often a "general frequency" axis and the interesting contrasts start at component two. This is the price of preserving sparsity, and it is usually worth paying.

LSA

Deerwester, Dumais, Furnas, Landauer and Harshman (1990), Indexing by latent semantic analysis, JASIS. Documents and terms embed into a shared k-dimensional space; similarity in that space handles synonymy (different words, same theme — recall improves) and partially handles polysemy (same word, several themes — each occurrence is forced to one point, its central weakness). Landauer and Dumais (1997) pushed the cognitive claim: LSA trained on textbook-scale corpora matches human synonym-test performance, an early hint that distributional statistics carry meaning — the intellectual ancestor of word2vec and modern embeddings.

Computation

For sparse X with nnz non-zeros, Lanczos/ARPACK methods compute k singular triplets in roughly O(nnz * k) plus O((n + d) k^2). Randomized SVD (Halko, Martinsson and Tropp, 2011) — sample the range with a random Gaussian matrix, orthonormalise, decompose the small projection — achieves near-optimal error with high probability in O(nnz * k) with better cache behaviour, and is sklearn's default. For streaming corpora, Gensim implements incremental one-pass LSA (Rehurek, 2010).

Successors

Probabilistic LSA (Hofmann, 1999) replaced the algebraic factorisation with a generative mixture; LDA (topic modelling territory; Blei et al., 2003) added Dirichlet priors; NMF swapped orthogonality for non-negativity and gained interpretability. For retrieval, dense neural embeddings have displaced LSA — but truncated SVD remains the workhorse for generic sparse-matrix compression: recommender matrices, graph adjacency spectra, and any place a 50,000-column sparse table needs to become 300 useful numbers.

What to learn next