Collaborative filtering

Setup#

pip install numpy

One library, no download, runs in milliseconds. A six-by-six table you can read with your eyes catches bugs that a hundred-thousand-row file hides completely.

User-based collaborative filtering, complete#

import numpy as np

USERS = ["Aarav", "Bhavna", "Chetan", "Divya", "Esha", "Farhan"]
FILMS = ["Sholay", "Lagaan", "3 Idiots", "Dangal", "Tumbbad", "Stree"]

# Ratings out of 5. A 0 means "not rated", NOT "hated it".
R = np.array([
    [5, 4, 5, 0, 1, 0],
    [4, 5, 4, 5, 2, 1],
    [5, 4, 0, 4, 1, 2],
    [1, 2, 1, 0, 5, 4],
    [2, 1, 0, 1, 4, 5],
    [0, 1, 2, 0, 5, 4],
], dtype=float)

SEEN = (R > 0).astype(float)
# Each person's own average, over the films they actually rated.
MEANS = R.sum(axis=1) / np.maximum(SEEN.sum(axis=1), 1)
C = (R - MEANS[:, None]) * SEEN          # unrated cells forced back to exactly 0


def user_similarity(C):
    norms = np.sqrt((C ** 2).sum(axis=1))
    norms[norms == 0] = 1.0
    S = (C @ C.T) / np.outer(norms, norms)
    np.fill_diagonal(S, 0.0)             # you are not your own neighbour
    return S


S = user_similarity(C)


def predict(u, i, k=2):
    """Predicted rating of film i for user u, from the k most similar people who rated it."""
    others = [v for v in range(len(USERS)) if SEEN[v, i] and S[u, v] > 0]
    others.sort(key=lambda v: -S[u, v])
    others = others[:k]
    if not others:
        return None
    w = np.array([S[u, v] for v in others])
    diffs = np.array([R[v, i] - MEANS[v] for v in others])
    return float(MEANS[u] + (w @ diffs) / w.sum())


print("who is most like Aarav?")
for v in sorted(range(len(USERS)), key=lambda v: -S[0, v]):
    if v != 0:
        print(f"  {S[0, v]:+.2f}  {USERS[v]}")

print("\nfilms Aarav has not rated:")
for i, film in enumerate(FILMS):
    if not SEEN[0, i]:
        p = predict(0, i)
        print(f"  {film:<9} predicted {p:.2f}" if p else f"  {film:<9} no evidence")

print("\nsame film, two very different people:")
for u in (0, 3):
    print(f"  Dangal for {USERS[u]:<7} {predict(u, 3):.2f}")

who is most like Aarav?
  +0.79  Chetan
  +0.48  Bhavna
  -0.42  Esha
  -0.70  Farhan
  -0.90  Divya

films Aarav has not rated:
  Dangal    predicted 4.81
  Stree     predicted 2.06

same film, two very different people:
  Dangal for Aarav   4.81
  Dangal for Divya   1.00

What that output is telling you#

The similarity column has split six people into two camps, and nobody supplied a genre. Chetan and Bhavna sit at +0.79 and +0.48. Divya sits at -0.90, which is close to a perfect opposite.

Negative similarity is real information, not noise. It says Divya's ratings move in the opposite direction to Aarav's.

Then look at the last two lines. The same film, Dangal, scores 4.81 for Aarav and 1.00 for Divya. One model, one table, two opposite answers. That is personalisation working.

Line by line, for the parts that are not obvious#

R.sum(axis=1) / np.maximum(SEEN.sum(axis=1), 1) — the average over rated films only, because zeros are absences. The np.maximum(..., 1) guards a user who has rated nothing, so you get a mean of 0.0 rather than a RuntimeWarning: invalid value encountered in divide and a table full of nan.

(R - MEANS[:, None]) * SEEN — the line the whole file depends on. Subtracting the mean makes ratings comparable across people. Multiplying by SEEN afterwards puts the unrated cells back to exactly zero. Without that second step, an unrated cell becomes a large negative number and reads as "hated it", which is exactly backwards.

(C @ C.T) / np.outer(norms, norms) — one matrix multiply computes every user against every other user. On mean-centred rows, this is the cosine similarity: a number from -1 to +1 saying whether two rating patterns point the same way. On mean-centred data over co-rated items it coincides with the Pearson correlation, which is why you see both names used for the same code.

MEANS[u] + (w @ diffs) / w.sum() — the prediction is not a weighted average of raw ratings. It is your baseline, plus a weighted average of how far your neighbours deviated from their baselines. Predicting raw ratings directly makes a harsh rater's list drift upwards, which is a common and hard-to-spot bug.

S[u, v] > 0 — only positive neighbours are allowed to vote. Somebody who is your opposite tells you what to avoid, and folding that into a weighted average with a negative weight makes the arithmetic unstable when weights nearly cancel.

Item-based, in three sentences#

Transpose the table and run the identical code. Rows become films, columns become people, and the output becomes film-to-film similarity.

Then score an unseen film by how similar it is to the films this person already rated highly. The full item-based version, with recommendations rather than predicted ratings, is written out in the recommendation system project.

Common mistakes#

Using zero as a rating. Every part of this file treats 0 as absent. Load a dataset where 0 is a legitimate rating and the code is silently wrong everywhere. Convert to NaN on load, or keep a separate mask, and never let the two meanings share a value.

Cosine on raw, uncentred ratings. Everybody's ratings are positive, so every pair of users looks similar and the similarity matrix collapses towards +1. You will get recommendations that are pure popularity wearing a costume.

Trusting similarity computed from two shared items. Two users who both rated exactly two films the same way get a similarity of 1.0, which is meaningless. Production systems apply significance weighting: multiply the similarity by min(n_common, 50) / 50, so thin overlaps are shrunk towards zero.

Forgetting to clip predictions. Nothing above constrains the output to the rating scale. A confident neighbourhood can predict 5.4 out of five. Wrap the return in min(5.0, max(1.0, value)) before it reaches a user interface.

Recomputing the whole similarity matrix on every request. It costs $O(|U|^2 |I|)$ and belongs in a nightly batch job, not a request handler. This is the practical reason item-based won.

Try it yourself#

Change k=2 to k=5 in predict and re-run. Aarav's Stree prediction will move, because weaker and more distant neighbours are now voting.

Then add a seventh user who has rated exactly one film. Watch what the similarity row for that person looks like, and decide for yourself whether you would show them anything at all.

What to learn next#

• Matrix factorisation — replaces the neighbourhood with learned features.
• Implicit feedback — the same problem when you have clicks and no ratings.
• The cold-start problem — the failure mode this method cannot escape.