Vector databases
A vector database stores text as numbers that capture meaning, then finds the closest matches to your question in milliseconds.
- 14 min read
- 3 reading levels
- Published
Read these first
On this page 10
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
The short answer
A vector database stores meaning as numbers, and finds the closest matches fast.
It is the search engine that sits underneath RAG.
The analogy you have already lived
Walk into a supermarket looking for soap. You do not read every label in the shop. You walk to the soap aisle, because similar things are kept near each other.
Somebody arranged that shop by meaning. Soaps together, snacks together, cold things at the back. Once the arrangement exists, finding something takes seconds instead of an hour.
A vector database does that arrangement for text. Sentences with similar meaning end up near each other. A search becomes a short walk to the right aisle.
Why word matching was not enough
Ordinary search matches letters. Type "reimbursed" and it looks for the letters r-e-i-m-b-u-r-s-e-d.
Your help page says "refund". Same meaning, different letters, zero matches. Now add the ways real people write: "money back", "paisa wapas", "cancel and return my payment". Keyword search fails on all of them.
The fix is to stop comparing spellings and start comparing meanings.
What a vector actually is
Every sentence gets turned into a long list of numbers. The thing that does this is an embedding model. Its only job is to place text at a position in an imaginary space.
You cannot picture that space, because it has hundreds of directions rather than three. But the rule is one you already understand from a map. Similar things sit close together. Different things sit far apart.
"Refund my money" and "I want to be reimbursed" land almost on top of each other. "Chicken biryani recipe" lands far away. Nobody wrote that down. The model learned it from reading enormous amounts of text.
How it works, in one picture
WHEN YOU ADD DOCUMENTS (once, in advance)
"Refunds reach your UPI app in five days"
|
v
[ embedding model ]
|
v
[ a long list of numbers ] -> stored, with the original text
WHEN SOMEONE ASKS (every time)
"when do I get my money back?"
|
v
[ embedding model ] -> [ a long list of numbers ]
|
v
[ vector database: which stored lists point the same way? ]
|
v
nearest: "Refunds reach your UPI app in five days"
next : "Cancelled orders are reversed automatically"Why it needs to be a database and not a loop
With a hundred documents you could compare against every one. With ten million, checking each one for every question is far too slow.
So a vector database builds an index in advance. An index is a shortcut structure, much like the one at the back of a textbook. It answers "what is near this?" without visiting everything.
There is a real trade here, and it is worth knowing before you pick a tool. That shortcut is approximate. It occasionally misses a true nearest neighbour in exchange for being hundreds of times faster. You choose where to sit on that trade.
Where you have already seen it
- Shopping apps showing "similar products" that share no words in the title.
- Photo apps letting you search "beach" without anyone tagging the photos.
- Music apps building a station from one song you liked.
- Any "chat with your documents" tool.
The honest part
Two things regularly surprise people.
The database only understands what the embedding model gave it. A weak or wrong-language embedding model cannot be fixed by a better database.
And embeddings are poor at exact identifiers. Order number 4417 and order number 4418 are close in meaning and completely different in fact. Serious systems run keyword search alongside vector search for this reason, and combine the two rankings.
Remember this
- Text becomes numbers, and nearby numbers mean similar meaning.
- The index trades a little accuracy for a very large amount of speed.
- Pair it with keyword search whenever exact codes and names matter.
What to learn next
- Embeddings — how text becomes those numbers.
- What is RAG? — what the search results are used for.
- Chat with your PDF — a full build.
Developer — Code and libraries.
Setup
pip install numpyA vector database is three ideas: a distance measure, an index that avoids scanning everything, and storage for the original text. The first is small enough to write by hand, and doing so makes every knob in a real system readable.
Similarity, by hand
We use three made-up meaning axes so every number is checkable.
import numpy as np
# Three made-up meaning axes: [about price, about phones, about food]
docs = {
"budget smartphone under 15000": [1.0, 1.0, 0.0],
"premium flagship phone": [0.0, 1.0, 0.0],
"cheap street food in Pune": [1.0, 0.0, 1.0],
}
names = list(docs)
M = np.array([docs[n] for n in names])
query = np.array([1.0, 1.0, 0.0]) # "cheap phone"
# Cosine similarity: 1.0 means the same direction, 0.0 means unrelated.
sims = (M @ query) / (np.linalg.norm(M, axis=1) * np.linalg.norm(query))
for rank, i in enumerate(np.argsort(-sims), start=1):
print(f"{rank}. {sims[i]:.2f} {names[i]}")1. 1.00 budget smartphone under 15000 2. 0.71 premium flagship phone 3. 0.50 cheap street food in Pune
The food document is not zero. It shares the "price" axis with the query, because "cheap" is a price word. That is a genuine property of embeddings, not an artefact of the toy: unrelated documents rarely score zero, so an absolute similarity threshold has to be tuned per model rather than guessed.
Why cosine and not plain distance
Cosine similarity measures the angle between two vectors and ignores their length. Length in an embedding tends to track how long or emphatic the text was, which is not what you want to search on.
There is a useful shortcut. If you normalise every vector to unit length first, the dot product is the cosine, and ranking by Euclidean distance gives the identical order. Most libraries normalise on insert for exactly this reason, and then use the cheapest operation available.
What brute force costs
Scanning every vector is exact and, up to a point, perfectly reasonable.
import numpy as np
rng = np.random.default_rng(0)
db = rng.normal(size=(50_000, 384)).astype("float32")
db /= np.linalg.norm(db, axis=1, keepdims=True) # unit length, so dot product = cosine
query = db[12345] # ask for a row we already know
scores = db @ query # 50,000 dot products in one call
print("vectors searched:", db.shape[0])
print("memory for the raw vectors:", round(db.nbytes / 1e6), "MB")
print("best match is the row we asked for:", int(scores.argmax()) == 12345)vectors searched: 50000 memory for the raw vectors: 77 MB best match is the row we asked for: True
Fifty thousand vectors is 77 MB and a single fast matrix multiply. Do not reach for a vector database at this size — NumPy is faster and has no operational cost.
Now scale it. Ten million vectors at 768 dimensions in float32 is about 30 GB, and every query touches all of it. That is the point where indexes and compression stop being optional.
With real embeddings
all-MiniLM-L6-v2 is around 90 MB and runs on CPU. The first call downloads it and prints progress bars.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
docs = ["How to reset your password",
"Our office is closed on Sunday",
"Recipe for masala chai"]
E = model.encode(docs, normalize_embeddings=True)
q = model.encode("I forgot my login password", normalize_embeddings=True)
print("shape of the document matrix:", E.shape)
print("closest document:", docs[int((E @ q).argmax())])shape of the document matrix: (3, 384) closest document: How to reset your password
The query and the winning document share no content words at all. "forgot my login password" against "reset your password" is exactly the match the keyword retriever in the RAG lesson could not make.
Only the winner is printed here, not the score. Similarity values shift between model versions, so a number printed in a lesson would go stale while the ranking stays stable.
Choosing a store
| Situation | Reasonable choice |
|---|---|
| Under about 100k vectors | NumPy in memory, or SQLite with a blob column |
| Single machine, needs persistence | FAISS, or an embedded store such as Chroma or LanceDB |
| Already running Postgres | pgvector, keeping filters and vectors in one place |
| Millions of vectors, many users | Qdrant, Weaviate, Milvus, or a managed service |
Start at the top of that table. Most projects that reach for a distributed vector service would be better served by the row above, and the migration path upward is short.
Common mistakes
Different embedding models for indexing and querying. The vectors then live in unrelated spaces and results are noise. Store the model name alongside the index and assert on it at query time.
Forgetting to normalise before using the dot product. Long documents win everything, because their vectors are longer. Normalise on insert.
Post-filtering instead of pre-filtering. Fetching the top 10 by similarity and then dropping everything outside category = "billing" can leave you with zero results. Use the store's native filtered search, which restricts the candidate set during the walk.
Ignoring updates and deletes. Graph indexes handle deletion by marking tombstones, and quality degrades as they accumulate. Plan a periodic rebuild before you need one.
Treating similarity as relevance. The nearest neighbour is always returned, even when nothing relevant exists. Add a threshold and a "nothing found" branch.
Try it yourself
Take the 50,000-vector example and time it with time.perf_counter. Then rerun with 500,000 vectors and time it again. Watch the cost grow in a straight line with the number of vectors. That straight line is the entire reason approximate indexes exist, and feeling it is more convincing than reading about it.
What to learn next
- Embeddings — picking and evaluating an embedding model.
- What is RAG? — using retrieved chunks in a prompt.
- LangChain — a common wrapper over these stores.
Researcher — Mathematics and papers.
Metric equivalences
For unit-normalised vectors a and b:
‖a − b‖² = ‖a‖² + ‖b‖² − 2·a·b = 2 − 2·cos(a,b)Squared Euclidean distance is a monotone decreasing function of cosine similarity, so the two induce the identical ranking. Inner product on un-normalised vectors does not: it is not a metric, it fails the triangle inequality, and index structures that assume metric-space properties can lose correctness guarantees under it. Maximum inner product search is normally reduced to nearest-neighbour search by an explicit transformation, or sidestepped by normalising at insert time.
The cost being avoided
Exact search is O(N · D) per query for N vectors of dimension D, with a memory footprint of 4ND bytes in float32. At N = 10⁷ and D = 768 that is roughly 7.7 × 10⁹ multiply-adds and about 30 GB resident. Approximate nearest neighbour methods trade a bounded loss in recall for orders of magnitude in both.
Index families
IVF (inverted file). Cluster the corpus into n_list cells by k-means and store each vector under its nearest centroid. At query time, probe the n_probe nearest cells. Expected scan is N · n_probe / n_list. Recall rises with n_probe, monotonically and with diminishing returns. Failure mode: a true neighbour sitting immediately across a cell boundary is missed unless n_probe is raised.
PQ (product quantisation). Split each D-dimensional vector into m sub-vectors, quantise each against a 256-entry codebook learned per subspace, and store one byte per sub-vector.
memory per vector: m bytes instead of 4D bytes
compression ratio: 4D / mFor D = 768 and m = 96, that is 96 bytes against 3072 bytes — a 32-fold reduction, turning 30 GB into under 1 GB. Distances are computed against the codebook with precomputed lookup tables, so the compressed representation is never decoded. Jégou et al. (2011) is the original. IVF-PQ is the workhorse combination in FAISS.
HNSW (hierarchical navigable small world). A multi-layer proximity graph. Each node keeps up to M neighbours per layer; upper layers are sparse and act as express lanes. Search greedily descends from an entry point, maintaining a candidate list of size efSearch. Construction is O(N log N); query is empirically O(log N). Malkov and Yashunin (2016) is the reference.
The three knobs behave predictably. M sets graph degree and therefore memory and recall ceiling. efConstruction sets build quality and build time. efSearch trades query latency against recall at run time, and is the only one adjustable without a rebuild. Memory is roughly 4ND + N · M · 2 · 4 bytes — the raw vectors plus neighbour lists, which is why HNSW is fast and expensive, and why HNSW over PQ codes exists.
ScaNN. Anisotropic vector quantisation (Guo et al., 2020), which weights quantisation error by its effect on inner product rather than minimising raw reconstruction error. Strong results on the recall-versus-throughput frontier.
Evaluating an index honestly
There is no single quality number. The correct object is a curve: recall@k against queries per second, swept over the index's tuning parameter, at a fixed memory budget. ann-benchmarks established this methodology and it is still the right one. A vendor benchmark quoting throughput without stating recall, or recall without stating memory, has omitted half the trade.
Filtered search
Combining metadata predicates with ANN is not a solved problem. Three approaches, each with a real failure case:
- Post-filter: search then filter. Can return fewer than
kresults, arbitrarily so under a selective predicate. - Pre-filter: build the candidate set from the predicate, then search within it. Exact, but degenerates to brute force when the filtered set is large.
- In-graph filtering: evaluate the predicate during traversal. Graph connectivity can break under a selective filter, stranding the search in a disconnected region.
Selectivity determines which is correct. Systems that pick one strategy statically perform badly across a realistic query mix.
Dimensionality and truncation
In high dimensions, distances between random points concentrate, and the contrast between nearest and farthest neighbour shrinks. This is the classic obstacle to nearest-neighbour search. Real embeddings are saved by lying on a much lower-dimensional manifold than their nominal dimension suggests.
Matryoshka representation learning (Kusupati et al., 2022) trains embeddings so that leading prefixes remain useful on their own. A 768-dimensional vector can be truncated to 128 dimensions with graceful degradation, which enables a cheap two-stage cascade: shortlist on truncated vectors, rerank on full ones.
Hybrid retrieval
Dense vectors underperform on exact identifiers, rare proper nouns and numeric codes, because embedding models are trained to place semantically similar things together and 4417 is semantically similar to 4418. Sparse lexical retrieval is exact on those cases and blind to paraphrase. Fusing the two ranked lists with reciprocal rank fusion (k = 60) avoids the score-calibration problem entirely and is close to free. Learned sparse representations such as SPLADE occupy a middle position, producing sparse vectors over vocabulary terms with learned expansion.
Papers
- Jégou et al., Product Quantization for Nearest Neighbor Search, IEEE TPAMI, 2011
- Malkov and Yashunin, Efficient and Robust ANN Search Using HNSW Graphs, 2016 — arxiv.org/abs/1603.09320
- Johnson et al., Billion-Scale Similarity Search with GPUs (FAISS), 2017 — arxiv.org/abs/1702.08734
- Guo et al., Accelerating Large-Scale Inference with Anisotropic Vector Quantization (ScaNN), 2020 — arxiv.org/abs/1908.10396
- Reimers and Gurevych, Sentence-BERT, 2019 — arxiv.org/abs/1908.10084
- Kusupati et al., Matryoshka Representation Learning, 2022 — arxiv.org/abs/2205.13147
- Muennighoff et al., MTEB: Massive Text Embedding Benchmark, 2022 — arxiv.org/abs/2210.07316
What to learn next
- What is RAG? — the retrieval pipeline these indexes serve.
- Embeddings — bi-encoder objectives and representation quality.
- AI agents — retrieval as one tool among several.