Learn AI Create the Future — with Pranay
Main site

Classical NLP That Still Works

BM25

BM25 ranks documents by how well their words match a search query, rewarding matches but refusing to keep rewarding a word that repeats over and over.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

BM25 scores how well a document matches a search query. It is the formula behind most search boxes you have ever typed into.

Think about judging a cricket player by runs scored. Going from 0 to 50 runs tells you a lot. Going from 250 to 300 tells you much less. The player has already proven themselves, and 50 more runs barely change your opinion.

BM25 treats word matches the same way. The first time a query word shows up in a document, it matters a lot. The tenth time, barely at all. That single idea is why BM25 has powered search engines for over thirty years.

Why it exists

TF-IDF rewards a document for containing a query word many times. But it does so without any limit. A document that repeats "chai" fifty times could outscore a genuinely useful page that mentions "chai" three times. That happens even if the first one is a spam page, stuffed with keywords.

TF-IDF also does not properly account for document length. A long document naturally contains more words, including more repeats of a query term. That happens purely by having more text, not because it is more relevant.

BM25 — the name stands for "Best Match 25", the 25th version tried during its development — fixes both problems at once.

How it works

Two ideas, stacked on top of TF-IDF's foundation.

Diminishing returns on repeats. The score for a word rises quickly at first, then flattens out. Ten repeats of "chai" scores noticeably higher than one repeat, but not ten times higher.

Length correction. A document is compared against the average document length in the whole collection. A short document that mentions "chai" once is judged more relevant than a long document mentioning "chai" once. In the short document, that one word is a bigger fraction of the whole thing.

   Query: "hot chai"

   Doc A (11 words):  "the chai stall serves hot chai every morning"
                        -> mentions BOTH query words, chai appears twice,
                           and the document is short -> high score

   Doc B (9 words):   "buy fresh coffee beans online for your kitchen"
                        -> mentions NEITHER query word -> score of zero

Search engines like Elasticsearch and OpenSearch use BM25, with these exact defaults, as their standard ranking formula.

Where you have already seen it

  • Every plain "search" box on the web. Product search, documentation search, forum search — none of it needing AI.
  • Elasticsearch and OpenSearch, the search engines behind countless apps, use BM25 by default.
  • The first stage of most modern AI search. Even systems using AI embeddings often run BM25 first, to shortlist candidates before a more expensive model re-ranks them.

Remember this

  • BM25 rewards word matches with diminishing returns. Stuffing a word in over and over stops helping after a point.
  • It corrects for document length, so short and long documents are judged fairly.
  • It remains, decades after being invented, the default ranking formula for most real-world search systems.

What to learn next

  • TF-IDF — the term-weighting idea BM25 builds on and refines.
  • Sentence-transformers — the modern, meaning-aware complement to BM25's word matching.
  • Vector databases — where the embedding half of a hybrid search system lives.

Developer — Code and libraries.

BM25 is a formula, not a training process. Building it from scratch, on a tiny set of documents, shows exactly what each part is doing.

Setup

Nothing to install — this uses only the standard library.

bash
python --version    # 3.9 or newer

BM25, implemented from scratch

bm25.py
import math
from collections import Counter

docs = [
    "the chai stall near the station serves hot chai every morning",
    "buy fresh coffee beans online, coffee delivered to your door",
    "our tea garden grows the finest tea in the hills",
    "chai, coffee and tea are the three most popular hot drinks in india",
]
tokenized = [d.split() for d in docs]
N = len(docs)
avgdl = sum(len(d) for d in tokenized) / N

doc_freq = Counter()
for d in tokenized:
    for term in set(d):
        doc_freq[term] += 1

k1, b = 1.5, 0.75  # standard defaults used by Elasticsearch and Lucene

def idf(term):
    n = doc_freq[term]
    return math.log((N - n + 0.5) / (n + 0.5) + 1)

def bm25_score(query, doc_tokens):
    freqs = Counter(doc_tokens)
    dl = len(doc_tokens)
    score = 0.0
    for term in query:
        if term not in freqs:
            continue
        f = freqs[term]
        num = f * (k1 + 1)
        den = f + k1 * (1 - b + b * dl / avgdl)
        score += idf(term) * (num / den)
    return score

query = "hot chai"
print(f"query: {query!r}")
print(f"avg doc length: {avgdl:.1f} words")
print()
scores = [(bm25_score(query.split(), d), doc) for d, doc in zip(tokenized, docs)]
for score, doc in sorted(scores, key=lambda x: -x[0]):
    print(f"  {score:.3f}  {doc}")
Output
query: 'hot chai'
avg doc length: 11.0 words

  2.413  the chai stall near the station serves hot chai every morning
  0.641  chai, coffee and tea are the three most popular hot drinks in india
  0.000  buy fresh coffee beans online, coffee delivered to your door
  0.000  our tea garden grows the finest tea in the hills

Line by line

Documents 2 and 4 score zero. Neither contains "hot" or "chai" at all — the if term not in freqs: continue line skips scoring for words the document does not have, and a document matching nothing scores exactly nothing.

Document 1 scores far above document 4, despite document 4 mentioning both "chai" and "hot" once each too. Document 1 mentions "chai" twice and is a below-average-length document, both of which BM25 rewards through the f (frequency) and dl / avgdl (length ratio) terms.

k1 = 1.5 controls how quickly term-frequency reward saturates. A higher k1 lets repeated terms keep contributing more; k1 = 0 would make BM25 ignore frequency entirely and only check presence or absence.

b = 0.75 controls how strongly length is corrected for. b = 1 fully normalizes for length; b = 0 disables length correction and BM25 degenerates toward plain frequency scoring.

Common mistakes

Reimplementing this per query instead of precomputing avgdl and document frequencies once. In the code above, doc_freq and avgdl are computed once, over the whole collection, before any query runs. Recomputing them per query is both slow and semantically wrong — those numbers describe the collection, not the query.

Forgetting that BM25 needs tokenized, normalized text. "Chai," with a trailing comma is a different string to Python than "chai". Real systems lowercase, strip punctuation and often stem or lemmatize before counting — see text normalization.

Assuming a higher BM25 score means "more relevant" in an absolute sense. BM25 scores are only meaningful relative to other documents scored against the same query and the same collection. Comparing a BM25 score from one search box to a BM25 score from a different website's search box tells you nothing.

Try it yourself

Change the query to "tea" alone and re-run. Document 3, which repeats "tea" twice in a short sentence, should now outscore document 4 by a wide margin, even though document 4 also mentions "tea" — a direct demonstration of the length-correction term at work.

What to learn next

  • TF-IDF — the term-weighting foundation BM25 is built on top of.
  • Dictionary matching — a different, exact-match approach to finding terms in text at speed.
  • What is RAG? — a common system that layers BM25-style retrieval underneath an LLM.

Researcher — Mathematics and papers.

The formula

text
score(D, Q) = sum over q in Q of  IDF(q) * ( f(q, D) * (k1 + 1) ) / ( f(q, D) + k1 * (1 - b + b * |D| / avgdl) )
  • D is a document, Q a query, q one query term.
  • f(q, D) is the raw frequency of q in D.
  • |D| is document length in tokens; avgdl the mean document length across the collection.
  • k1 (typically 1.2–2.0) controls term-frequency saturation.
  • b (typically 0.75) controls the strength of length normalization, 0 <= b <= 1.

IDF(q), in the Robertson-Sparck Jones probabilistic form:

text
IDF(q) = ln( (N - n(q) + 0.5) / (n(q) + 0.5) + 1 )
  • N is total document count, n(q) the number of documents containing q.
  • The +0.5 terms are a continuity correction inherited from the underlying probabilistic model, keeping the estimate well-behaved when n(q) is small or close to N.

Where the formula comes from

BM25 derives from the Probabilistic Relevance Framework (Robertson & Spärck Jones, 1976), which models retrieval as estimating P(relevance | D, Q) via a binary independence model over term presence. The 2-Poisson model (Bookstein & Swanson, 1974) — assuming term frequency within a document follows one Poisson distribution for "relevant" documents and another for "non-relevant" ones — motivates the specific saturating shape of the f * (k1+1) / (f + k1(...)) term: it behaves near-linearly for small f and asymptotically approaches k1 + 1 as f -> infinity, which is exactly the diminishing-returns curve.

BM25 itself, as an approximation to the 2-Poisson model tractable enough to compute at scale, was developed by Robertson, Walker and colleagues through the Okapi system at TREC-3 (Robertson & Walker, 1994), after testing (hence "25") many variant weighting schemes.

Complexity

Scoring one query against N documents in an inverted-index system is O(sum over q in Q of |posting list(q)|) — proportional only to the number of documents actually containing at least one query term, not to N. This sub-linear-in-collection-size behaviour, inherited from the underlying inverted index rather than from BM25 itself, is what makes BM25 practical at web scale.

Relationship to TF-IDF

BM25 reduces to something close to classical TF-IDF as k1 -> infinity (no saturation) and b = 0 (no length normalization). BM25's two additional parameters are precisely the two fixes: bounded term-frequency reward, and length-aware normalization, both grounded in the probabilistic model rather than added as ad hoc heuristics.

Known limitations

BM25 has no notion of term proximity or word order in its base form — "hot chai" and "chai hot" score identically, and a document containing "hot" and "chai" at opposite ends scores the same as one where they are adjacent, unless a proximity-aware variant is used. It also treats every query term's contribution as additive and independent, so it cannot represent that "New York" together means something different from "New" and "York" scored separately — the same limitation TF-IDF has, inherited directly.

Key references

  • Bookstein, A. & Swanson, D. (1974). Probabilistic Models for Automatic Indexing. Journal of the American Society for Information Science.
  • Robertson, S. E. & Spärck Jones, K. (1976). Relevance Weighting of Search Terms. Journal of the American Society for Information Science 27(3), 129–146.
  • Robertson, S. E., Walker, S., et al. (1994). Okapi at TREC-3. Text REtrieval Conference. First large-scale evaluation of BM25.
  • Robertson, S. & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3(4), 333–389. The definitive modern survey.

Current state and open problems

BM25 remains, as of the current generation of production search systems, the default first-stage retrieval method even in pipelines that finish with a neural re-ranker or a large language model — it is cheap, has no training cost, and its failure modes (pure lexical mismatch, no semantic generalization) are well understood and can be compensated for downstream. The dominant architecture in modern search and retrieval-augmented systems is hybrid retrieval: BM25 and a dense embedding retriever run in parallel, and their results are merged, often with reciprocal rank fusion (Cormack, Clarke & Buettcher, 2009), because the two methods make different, largely uncorrelated errors — BM25 misses paraphrases and synonyms, dense retrieval sometimes drifts toward vaguely-related-but-wrong matches on short or ambiguous queries.

What to learn next

  • Sentence-transformers — the dense-retrieval half of a modern hybrid search system.
  • Vector databases — how BM25 and dense embeddings are combined in a production hybrid search system.
  • TF-IDF — the simpler ancestor BM25 refines.