Classical NLP That Still Works
Bag of words
Bag of words turns a sentence into a list of word counts, throwing away word order but keeping enough signal to search and classify text cheaply.
- 9 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Bag of words counts how many times each word appears in a piece of text. It ignores the order they came in.
Picture emptying your kitchen shelf into one big bag. Rice, dal, salt, three kinds of spice, all mixed together. You can no longer tell which shelf a packet sat on, or what order you put them in. But you can still count: two packets of rice, one salt, three spice jars.
That count alone tells you a lot about what kind of kitchen this is. Bag of words does the same thing to a sentence. It throws away the order of words and keeps only the counts.
Why it exists
Computers are built to work with numbers, not sentences. A computer cannot compare, sort or classify raw text on its own. It needs the text turned into numbers first.
The earliest, simplest fix: count words. Build a giant list of every word your text might use — the vocabulary, a numbered list of every distinct word. Then, for each sentence, write down how many times each vocabulary word showed up.
That list of counts is a vector — a list of numbers, one per vocabulary word. A computer can compare, sort and do maths on it. This idea is decades old. It still runs quietly inside search engines, spam filters and recommendation systems today, because it is fast and it works.
How it works
Say your entire vocabulary, across every sentence you care about, is six words.
Vocabulary: [ chai, coffee, hot, sweet, the, was ]
Sentence: "the chai was hot"
Count each vocabulary word in the sentence:
chai -> 1
coffee -> 0
hot -> 1
sweet -> 0
the -> 1
was -> 1
Vector: [ 1, 0, 1, 0, 1, 1 ]Every sentence becomes a row of numbers the same length as the vocabulary. Most of those numbers are zero, because most sentences use only a handful of the words your vocabulary knows about.
Two sentences that share more words end up with more similar vectors. That similarity is the entire trick. A search engine matching your query to a webpage is often doing nothing fancier than comparing count vectors. Same for a spam filter deciding if an email is junk.
Where you have already seen it
- Old-school spam filters. Emails full of "free", "winner", "click" got flagged by counting those words, no understanding of the sentence needed.
- Document search before Google got smart. Early search engines matched query words against page word-counts.
- "People who bought this also bought." Some recommendation systems compare products by counting shared words in their descriptions.
Remember this
- Bag of words turns a sentence into a vector of word counts, and drops word order entirely.
- Two sentences with similar counts are treated as similar, even if their meaning is not.
- It is old, cheap and still a reasonable first thing to try on any new text problem.
What to learn next
- TF-IDF — fixing bag of words' biggest weakness: common words drowning out the useful ones.
- Tokenization — deciding what counts as a "word" in the first place.
- Embeddings — the modern replacement that keeps meaning, not only counts.
Developer — Code and libraries.
One short program shows what bag of words actually produces, and why throwing away word order is a real cost, not a footnote.
Setup
pip install scikit-learnCounting words with scikit-learn
from sklearn.feature_extraction.text import CountVectorizer
docs = [
"the chai was hot",
"the coffee was hot",
"the chai was sweet",
]
vec = CountVectorizer()
X = vec.fit_transform(docs)
print("vocabulary:", vec.get_feature_names_out())
print()
print("matrix:")
print(X.toarray())vocabulary: ['chai' 'coffee' 'hot' 'sweet' 'the' 'was'] matrix: [[1 0 1 0 1 1] [0 1 1 0 1 1] [1 0 0 1 1 1]]
Line by line
get_feature_names_out() returns the vocabulary in alphabetical order. Position 0 is always "chai" for this vocabulary, no matter which document you look at. That fixed ordering is what makes rows comparable to each other.
Row 0 is [1, 0, 1, 0, 1, 1]. Read it against the vocabulary: one "chai", zero "coffee", one "hot", zero "sweet", one "the", one "was". That is exactly "the chai was hot".
Rows 0 and 1 differ in exactly one position — "chai" versus "coffee". Everything else about the two sentences matches. Bag of words correctly notices these two sentences are structurally almost identical.
The word-order problem
from sklearn.feature_extraction.text import CountVectorizer
docs = [
"the dog bit the man",
"the man bit the dog",
]
vec = CountVectorizer()
X = vec.fit_transform(docs)
print("vocabulary:", vec.get_feature_names_out())
print(X.toarray())
print("rows identical:", (X.toarray()[0] == X.toarray()[1]).all())vocabulary: ['bit' 'dog' 'man' 'the'] [[1 1 1 2] [1 1 1 2]] rows identical: True
"The dog bit the man" and "the man bit the dog" describe opposite events. Bag of words produces the exact same vector for both. This is not a rare edge case — it is the method's core limitation, and it is why bag of words alone is a weak choice whenever word order changes meaning.
Common mistakes
Forgetting to fit and transform on the same vocabulary. vec.fit_transform(train_docs) learns the vocabulary and encodes it in one step. New text at prediction time must use vec.transform(new_docs), never fit_transform again — refitting builds a different vocabulary, and your numbers stop lining up with your trained model.
Not removing extremely common words. Words like "the" and "was" appear in almost every sentence and add little signal while inflating the vector. CountVectorizer(stop_words="english") removes a standard list of these. See stopwords for when this helps and when it quietly deletes useful information.
Assuming a bigger vocabulary is always better. A vocabulary built from millions of documents can reach hundreds of thousands of words. Most entries in most rows are zero. This is called a sparse matrix — mostly empty — and it costs memory and time even though it carries little information per cell.
Try it yourself
Add ngram_range=(1, 2) to CountVectorizer and re-run the dog/man example. Word pairs like "dog bit" and "bit man" become vocabulary entries too, so word order starts to leave a trace. The two sentences will no longer be identical — check which pairs differ.
What to learn next
- TF-IDF — reweighting these counts so rare, informative words count for more.
- BM25 — the ranking formula search engines actually use on top of these counts.
- Named entity recognition — a task where word order and position matter, and bag of words alone falls short.
Researcher — Mathematics and papers.
The vector space model
Bag of words is the concrete instance of the vector space model for text (Salton, Wong & Yang, 1975), where a document d is represented as a vector in R^|V|, V the vocabulary:
d = (c_1, c_2, ..., c_|V|)c_iis the raw count of vocabulary termiin documentd.
Similarity between two documents is typically cosine similarity:
sim(d_1, d_2) = (d_1 . d_2) / (||d_1|| * ||d_2||)d_1 . d_2is the dot product of the two count vectors.||d||is the Euclidean norm ofd.
Cosine, not Euclidean distance, is used because it is invariant to document length — a long document repeating the same topic should not automatically look "far" from a short document on the same topic.
What is provably lost
Bag of words is a multiset representation: it retains term frequency but discards position. Formally, it is invariant under any permutation pi of the token sequence — bow(pi(d)) = bow(d) for every permutation pi. Any distinction that depends on order — negation scope, syntactic role, coreference — is unrecoverable from d alone. This is not a limitation of a particular implementation; it is a property of the representation itself, and no amount of data changes it.
n-gram extensions (ngram_range=(1, k) in scikit-learn) partially restore local order by adding contiguous k-token spans as extra vocabulary items, at a cost: vocabulary size grows roughly as O(|V|^k) in the worst case, and most n-grams for n >= 3 occur once or never in a finite corpus, so their statistics are unreliable. This is the same sparsity-versus-context trade-off that appears in n-gram language models.
Complexity
For a corpus of D documents with total token count T and vocabulary size V: building the count matrix is O(T). The resulting D x V matrix is stored in sparse format (CSR in scikit-learn), with storage O(nnz) where nnz is the number of non-zero entries — in practice close to T, far below D * V.
Where bag of words still wins
Empirically, on tasks with clear lexical signal — spam detection, coarse topic classification, language identification — a bag-of-words-plus-linear-classifier pipeline is a strong, fast baseline that is hard for a large neural model to beat by a wide margin without substantially more compute. See the baseline you must beat before reaching for BERT for a measured comparison.
Key references
- Luhn, H. P. (1957). A Statistical Approach to Mechanized Encoding and Searching of Literary Information. IBM Journal of Research and Development. Early statistical text-counting for information retrieval.
- Salton, G., Wong, A. & Yang, C. S. (1975). A Vector Space Model for Automatic Indexing. Communications of the ACM 18(11), 613–620.
- Harris, Z. S. (1954). Distributional Structure. Word 10(2-3), 146–162. The distributional-semantics idea that both bag of words and modern embeddings ultimately trace back to.
Current state and open problems
Bag of words is not a research frontier — it is a settled, well-understood baseline. Its role today is as the floor everything else must clear: text-classification-baselines documents by how much, and how cheaply. Where it remains genuinely useful is in extremely high-throughput, latency-sensitive settings — first-stage filters, cheap deduplication, offline batch scoring — where a V-dimensional sparse dot product beats a neural forward pass by orders of magnitude and the accuracy gap does not matter enough to pay for it.
What to learn next
- TF-IDF — the term-weighting refinement that made vector-space retrieval practical.
- Word2Vec — replacing sparse counts with dense, learned vectors.
- Curse of dimensionality — why very high-dimensional sparse vectors behave strangely.