Learn AI Create the Future — with Pranay
Main site

Classic Algorithms in Depth

Naive Bayes

Naive Bayes classifies by adding up the evidence each clue carries, pretending the clues are unrelated — a wrong assumption that works absurdly well on text.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Naive Bayes classifies things by adding up the evidence from each clue, treating every clue as if it were independent of the others.

Think of sorting the post at home. An envelope arrives: glossy paper, bright red print, the words "WINNER" and "FREE" through the window, no handwriting anywhere. Each clue on its own nudges you toward "advertisement". Together, the nudges pile up into near-certainty — you bin it unopened.

Naive Bayes scores messages the same way. Each word in an email nudges the verdict toward "spam" or "normal", and the nudges add up.

Why it exists

To sort email perfectly you would need to know how every word interacts with every other word. "Free" inside "free entry — win big" is spam; "free" inside "are you free tomorrow?" is not. Modelling all word combinations is hopeless: there are more combinations than emails ever written.

Naive Bayes makes a bold simplification: pretend each word gives its evidence independently, as if the other words were not there. That pretence is the "naive" part — it is factually wrong, since words travel in packs.

And yet the sums land on the right side remarkably often. To pick the winning label you need the evidence ranking, not a perfect model of language. Wrong-but-useful is the entire character of this algorithm.

How it works

During training, the model counts words: how often "win" appears in spam versus normal mail, and so on for every word.

new message: "win a free prize"

  "win"   → seen mostly in spam     → nudge toward SPAM
  "free"  → seen mostly in spam     → nudge toward SPAM
  "prize" → seen mostly in spam     → nudge toward SPAM
  "a"     → seen everywhere         → no nudge

  add the nudges  →  SPAM, with high confidence

A real example you have seen

The spam folder in Gmail's early years ran on exactly this idea, and lightweight mail filters still do. It is also behind quick language detectors. The character patterns of Hindi, Tamil and English each leave distinctive counts. News apps use it too, sorting articles into sport, politics and business.

Remember this

  • Naive Bayes counts how often each clue appears with each label, then adds up evidence.
  • The "naive" part: it pretends clues are independent, which is false but works.
  • It trains in one fast pass, needs little data, and remains a strong first model for text.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn

Outputs verified with scikit-learn 1.7.2 on CPU.

A six-message spam filter

spam_filter.py
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

texts = ["win cash prize now", "lowest price guaranteed win big",
         "free entry win lottery", "are we meeting tomorrow",
         "send me the class notes", "lunch at the canteen tomorrow"]
labels = [1, 1, 1, 0, 0, 0]        # 1 = spam, 0 = normal

vec = CountVectorizer()
X = vec.fit_transform(texts)
print("vocabulary size:", len(vec.vocabulary_))

model = MultinomialNB().fit(X, labels)
for message in ("win a free prize", "send the notes tomorrow"):
    p = model.predict_proba(vec.transform([message]))[0]
    print(f"{message!r}: P(spam) = {p[1]:.3f}")
Output
vocabulary size: 23
'win a free prize': P(spam) = 0.946
'send the notes tomorrow': P(spam) = 0.030

Six training messages. That is all it took to separate these two confidently — the data-efficiency that keeps Naive Bayes alive.

The walkthrough

CountVectorizer turns each message into word counts over the 23-word vocabulary — the bag-of-words representation, where word order is thrown away. "win a free prize" becomes a mostly-zero row with ones under win, free and prize.

MultinomialNB is the variant for count data. Training is counting: how often each vocabulary word occurs in each class, plus how common each class is. One pass over the data, no iterations, no learning rate. There is no faster training in machine learning.

The word "a" is missing from the vocabulary. CountVectorizer drops single-letter tokens by default. Unseen and dropped words contribute nothing — the model scores only the evidence it recognises.

Why not 1.000 and 0.000? Every word keeps a tiny count in both classes thanks to smoothing (below), so no single word can be infinitely decisive. 0.946 is strong evidence, not a verdict.

alpha, the smoothing knob. Suppose "lottery" never appeared in a normal message. Raw counting would then assign probability zero — one word would veto the class no matter what forty other words say. Laplace smoothing (alpha=1.0, the default) adds a phantom count to every word-class pair, keeping every probability above zero.

Common mistakes

Feeding TF-IDF or negative values into MultinomialNB. The multinomial model is a story about counts. It tolerates TF-IDF in practice but the maths no longer means what it says, and negative values raise an error outright. For real-valued features use GaussianNB.

Trusting the probabilities. The 0.946 above ranks correctly, but Naive Bayes probabilities are notoriously overconfident — double-counted evidence from correlated words ("free" and "prize" travel together) piles up. Rank with them, threshold with care, calibrate if downstream decisions need honest numbers.

Forgetting the shared vocabulary. Calling vec.fit_transform again on new messages builds a new vocabulary that scrambles every column. Fit the vectoriser once; afterwards, transform only.

Wrong variant for the data. MultinomialNB for counts, BernoulliNB for present/absent flags, GaussianNB for continuous measurements. Mixing these up quietly degrades accuracy.

Try it yourself

Add a training message that puts "win" into a normal context — "did we win the match" with label 0 — then re-score "win a free prize". Predict the direction of change before running. Then set alpha=0.00001 and observe how the probabilities harden toward 0 and 1.

What to learn next

Researcher — Mathematics and papers.

The model

Bayes' rule for a class $c$ and feature vector $x = (x_1, \dots, x_d)$:

$$ P(c \mid x) = \frac{P(c) \, P(x \mid c)}{P(x)} $$

The naive assumption factorises the likelihood:

$$ P(x \mid c) = \prod_{j=1}^{d} P(x_j \mid c) $$

Where:

  • $P(c)$ — the prior: the class frequency in training data.
  • $P(x_j \mid c)$ — per-feature likelihood, estimated by counting.
  • $P(x)$ — the evidence; constant across classes, so ignored when ranking.
  • $d$ — vocabulary size (for text).

Classification takes the maximum a posteriori class, computed in log space to avoid underflow:

$$ \hat{c} = \arg\max_c \; \log P(c) + \sum_{j=1}^{d} x_j \log \theta_{jc} $$

with $\theta_{jc}$ the smoothed probability of word $j$ in class $c$ and $x_j$ its count. Laplace smoothing with parameter $\alpha$:

$$ \theta_{jc} = \frac{N_{jc} + \alpha}{N_c + \alpha d} $$

where $N_{jc}$ counts word $j$ in class $c$ and $N_c = \sum_j N_{jc}$. This is the posterior mean under a symmetric Dirichlet prior — smoothing is Bayesian estimation wearing overalls.

Naive Bayes is a linear model

The log-posterior above is linear in the count vector $x$: the score for class $c$ is $b_c + w_c^\top x$ with $w_{jc} = \log \theta_{jc}$. Naive Bayes and logistic regression therefore share a hypothesis class and differ only in fitting: generative counting versus discriminative maximum likelihood. Ng and Jordan (2001), On discriminative vs. generative classifiers, formalise the trade: Naive Bayes converges to its (higher) asymptotic error with $O(\log d)$ samples, logistic regression to its lower one with $O(d)$ — the small-data advantage seen in the demo.

Why wrong independence still ranks well

Domingos and Pazzani (1997) show zero-one loss depends only on the argmax staying correct: badly mis-estimated posteriors are harmless while the winner is preserved, and they characterise settings (including some with fully dependent features) where the naive classifier remains Bayes-optimal. The flip side: probability estimates are systematically extreme — hence calibration before thresholding.

Complexity

Training: one counting pass, $O(N_{tokens})$; memory $O(cd)$ for $c$ classes. Prediction: $O(\text{non-zero features})$ per document with sparse vectors. This puts Naive Bayes among the few classifiers whose training is effectively free at any scale.

Variants and standing

Event models: multinomial vs. Bernoulli analysed in McCallum and Nigam (1998). Complement NB (Rennie et al., 2003, Tackling the poor assumptions of naive Bayes text classifiers) fixes skewed-class bias and ships in scikit-learn as ComplementNB. As a baseline, NB+bigrams remains embarrassingly competitive on short-text classification, and "NB-SVM" hybrids (Wang and Manning, 2012) were state of the art on sentiment tasks into the deep-learning era.

What to learn next