Learn AI Create the Future — with Pranay
Main site

Classic Algorithms in Depth

Distance metrics

Euclidean, Manhattan and cosine measure "how similar" in different ways, and swapping one for another can change what your model believes.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A distance metric is the rule a model uses to decide how similar two things are.

Think about travelling across a city. A crow flies straight over the rooftops. An auto-rickshaw must follow the streets, block by block, turn by turn. Both measure the trip between the same two points — and they report different numbers.

Models face the same choice. Before any algorithm can say "these two customers are alike", someone must pick the measuring rule.

Why it exists

Every "find similar things" system — K-nearest neighbours, clustering, recommendations, search — rests on a similarity measure. There is no single correct one.

The crow's straight line is called Euclidean distance. The rickshaw's street-by-street total is called Manhattan distance, named after New York's grid of blocks.

And sometimes distance itself is the wrong question. Two restaurant reviews might use the same words in the same proportions, but one is three lines and one is three pages. Length makes them far apart. Direction makes them alike. Cosine similarity compares only the direction — what the review is about, not how long it rambles.

How it works

              crow (Euclidean):    straight line, corner to corner
   A ─────╮
   │      │   rickshaw (Manhattan): along the streets, add up the blocks
   ╰──────╯B
              cosine:  ignore the trip — do A and B point the same way?

A real example you have seen

When a music app says two songs are similar, each song is a long list of numbers describing its sound. Whether the app compares those lists with straight-line distance or with direction decides which song plays next. Search engines matching your query to pages lean on cosine, because a ten-word query must match a thousand-word page.

Remember this

  • "Similar" is not one idea — Euclidean, Manhattan and cosine measure different things.
  • Cosine ignores size and compares direction — perfect when lengths differ wildly.
  • Changing the metric changes the model's opinion, without touching the data.

What to learn next

  • Support vector machines — a model built on dot products, ready for the kernel trick.
  • Embeddings — learned vectors where cosine similarity means meaning.
  • Clustering — grouping data, with distance choice deciding the groups.

Developer — Code and libraries.

Setup

bash
pip install scikit-learn

Outputs verified with scikit-learn 1.7.2 on CPU.

Three reviews, three verdicts

distances.py
import numpy as np
from sklearn.metrics.pairwise import pairwise_distances, cosine_similarity

# word counts across three reviews: [good, service, slow]
a = np.array([[2, 1, 0]])    # short happy review
b = np.array([[20, 10, 0]])  # long review, same opinion
c = np.array([[0, 1, 2]])    # short unhappy review

for name, u, v in [("a vs b", a, b), ("a vs c", a, c)]:
    print(name,
          "| euclidean", round(pairwise_distances(u, v)[0, 0], 2),
          "| manhattan", round(pairwise_distances(u, v, metric="manhattan")[0, 0], 2),
          "| cosine similarity", round(cosine_similarity(u, v)[0, 0], 3))
Output
a vs b | euclidean 20.12 | manhattan 27.0 | cosine similarity 1.0
a vs c | euclidean 2.83 | manhattan 4.0 | cosine similarity 0.2

Read that carefully. Euclidean says review a sits 7 times closer to the unhappy review c than to b — a review with the identical opinion. Cosine says a and b match perfectly.

The walkthrough

Why Euclidean gets fooled here. Review b uses the same words as a, ten times over. Those extra repetitions stretch the straight-line gap to 20.12. Word counts grew, opinion did not.

Why cosine gets it right. a is [2, 1, 0] and b is [20, 10, 0] — one is the other times ten, so they point in exactly the same direction. Cosine similarity hits 1.0, its maximum. For a vs c the shared word "service" leaves a weak overlap of 0.2.

Note the units. Distance means smaller is closer. Similarity means bigger is closer. Mixing those up inverts every ranking downstream, and nothing crashes to warn you.

Plugging a metric into a model is one argument — and it can change the answer:

metric_changes_the_verdict.py
from sklearn.neighbors import KNeighborsClassifier

X = np.vstack([b, c])          # b is the happy review, c the unhappy one
y = np.array([1, 0])           # 1 = happy, 0 = unhappy

for metric in ("euclidean", "manhattan", "cosine"):
    model = KNeighborsClassifier(n_neighbors=1, metric=metric).fit(X, y)
    print(f"{metric:10s} -> review a labelled {model.predict(a)[0]}")
Output
euclidean  -> review a labelled 0
manhattan  -> review a labelled 0
cosine     -> review a labelled 1

Same three reviews, same model, one keyword changed — and the verdict flips.

Manhattan is often steadier than Euclidean when features contain outliers, because differences are not squared before adding.

Common mistakes

Comparing raw counts with Euclidean. As above — long documents drift far from short ones regardless of content. Use cosine, or normalise each row to length 1 first (then Euclidean ranks identically to cosine).

Skipping feature scaling. A salary column in thousands flattens an age column in tens, whatever metric you choose. Scale first; the KNN lesson shows the verdict flipping.

Cosine on data that is not centred. Cosine cares about direction from the origin (the all-zeros point). For word counts the origin is meaningful — an empty document. For arbitrary sensor data it may not be, and cosine can mislead. Centre the data or think twice.

Euclidean on one-hot categories. Two different cities encoded one-hot always sit the same distance apart — Delhi vs Mumbai equals Delhi vs Pune. The metric is not wrong, but it cannot express "more similar city". Learned embeddings exist for exactly this reason.

Try it yourself

Add review d = [[4, 2, 0]] — twice review a. Before running, predict its cosine similarity with a, then check its Euclidean distance too. Then normalise all four reviews to unit length and confirm Euclidean now agrees with cosine's ranking.

What to learn next

  • Support vector machines — a model built on dot products, ready for the kernel trick.
  • Embeddings — learned vectors where cosine similarity means meaning.
  • Clustering — grouping data, with distance choice deciding the groups.

Researcher — Mathematics and papers.

The Minkowski family

For $x, y \in \mathbb{R}^d$, the Minkowski distance of order $p \geq 1$:

$$ D_p(x, y) = \left( \sum_{i=1}^{d} |x_i - y_i|^p \right)^{1/p} $$

Where:

  • $d$ — number of features; $x_i, y_i$ — the $i$-th coordinates.
  • $p = 1$ — Manhattan; $p = 2$ — Euclidean; $p \to \infty$ — Chebyshev, $\max_i |x_i - y_i|$.

A metric formally requires non-negativity, identity of indiscernibles, symmetry and the triangle inequality: $D(x, z) \leq D(x, y) + D(y, z)$. Minkowski satisfies all four for $p \geq 1$.

Cosine similarity is not a metric:

$$ S_{\cos}(x, y) = \frac{x \cdot y}{\lVert x \rVert \, \lVert y \rVert} $$

with $x \cdot y$ the dot product and $\lVert x \rVert$ the Euclidean norm. The quantity $1 - S_{\cos}$ violates the triangle inequality, though $\sqrt{2(1 - S_{\cos})}$ — Euclidean distance between the unit-normalised vectors — restores it. Libraries that index "cosine distance" exploit exactly this equivalence.

Mahalanobis and learned metrics

Euclidean treats every direction as equally important. The Mahalanobis distance corrects for feature correlation and scale:

$$ D_M(x, y) = \sqrt{(x - y)^\top \Sigma^{-1} (x - y)} $$

with $\Sigma$ the covariance matrix of the data. It is Euclidean distance in the whitened space $\Sigma^{-1/2}x$.

Generalising $\Sigma^{-1}$ to any learned positive semi-definite matrix gives metric learning: LMNN (Weinberger and Saul, 2009) optimises the matrix so same-class points pull together and different-class points push apart. The modern descendant is deep metric learning — contrastive and triplet losses (Schroff et al., 2015, FaceNet) train an embedding network so that plain cosine or Euclidean in embedding space encodes semantic similarity. Sentence-transformer retrieval works this way.

Distance concentration

For i.i.d. coordinates, as $d \to \infty$ the ratio between nearest and farthest neighbour distances tends to 1 (Beyer et al., 1999). Fractional norms ($p < 1$, no longer metrics) concentrate more slowly (Aggarwal et al., 2001), but the practical remedy is reducing dimension or learning an embedding, not exotic norms.

Cost

All pairwise formulas above are $O(d)$ per pair, $O(n^2 d)$ for a full matrix. Mahalanobis adds a one-off $O(d^3)$ inversion. pairwise_distances computes blockwise with BLAS; for $n$ in the millions, approximate indexes (HNSW, IVF-PQ) replace exact computation.

What to learn next

  • Support vector machines — a model built on dot products, ready for the kernel trick.
  • Embeddings — learned vectors where cosine similarity means meaning.
  • Clustering — grouping data, with distance choice deciding the groups.