Learn AI Create the Future — with Pranay
Main site

Vision Datasets and Annotation

Deduplicating an image dataset

Web-collected image sets are full of near-copies, and finding them needs a hash that survives re-compression rather than a hash of the raw bytes.

Read these first

On this page 10
  1. The short answer
  2. The analogy
  3. Why duplicates hurt
  4. Two kinds of hash
  5. How you compare them
  6. What it catches and what it misses
  7. Choosing the cut-off
  8. Where you have seen this
  9. Remember this
  10. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Two pictures can look identical and be completely different files, so compare what they look like.

The analogy

Think about a family photo passed around on messaging apps.

Your cousin sends it to you. You forward it to an uncle. He posts it in a group. By the third hop the file is smaller, slightly softer, and possibly cropped a little.

Every one of those is the same photograph. Not one of them is the same file.

Now imagine collecting a thousand photos from those groups. You have far fewer distinct pictures than you think.

Why duplicates hurt

They waste labelling money. Somebody labels the same picture five times and you pay five times.

They distort what your model learns. A picture appearing fifty times is worth fifty examples during training. Your model becomes very good at that one picture.

They break your testing. This is the serious one. Suppose a picture is in your training set and a near-copy is in your test set. The model has seen the answer. Your test score is inflated and you do not know by how much.

That last problem has a whole lesson of its own, and it follows this one.

Two kinds of hash

A hash is a short code computed from a file. Same input, same code.

The ordinary kind is computed from the raw bytes. Change one byte and the code changes completely. That catches only files that are bit-for-bit identical.

The other kind is computed from what the picture looks like. Shrink it, throw away the fine detail, and describe the rough pattern of light and dark. Two versions of the same photo produce almost the same code, even after re-saving.

That second kind is called a perceptual hash.

   ordinary hash:   re-save the same photo -> completely different code
   perceptual hash: re-save the same photo -> same or nearly same code

How you compare them

Perceptual codes are compared by counting how many positions differ.

Zero differences means the codes are the same. A few differences means very similar. Many differences means different pictures.

You pick a cut-off. Below it, treat as duplicates. Above it, treat as distinct.

What it catches and what it misses

You will see this measured next, and the results are clear.

Perceptual hashing catches re-compression, resizing and brightness changes. Those are the common cases, and it handles them very well.

It misses mirror images, rotations and crops. A picture flipped left to right looks the same to you and scores as a completely different picture.

For those you need a different tool, one that compares meaning rather than pattern.

Choosing the cut-off

There is no universal right number. Set it too low and you miss duplicates. Set it too high and you throw away genuinely different pictures.

The only honest way is to build a small set where you know the answer, and try several cut-offs. The next section shows exactly that table.

Where you have seen this

  • Photo apps offering to clear duplicate pictures from your phone.
  • Reverse image search finding a picture that has been resized.
  • Music apps recognising a song from a noisy recording. Same idea, different medium.

Remember this

  • A byte hash catches only identical files, which is a small fraction of duplicates.
  • A perceptual hash survives re-saving, resizing and brightness changes.
  • It misses flips, rotations and crops. Know that before you trust it.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install numpy==1.26.4 scipy==1.14.1 pillow==11.0.0

A perceptual hash from scratch, and what it survives

phash.py
import numpy as np, io, hashlib
from scipy.fft import dctn
from PIL import Image

rng = np.random.default_rng(0)
y, x = np.mgrid[0:256, 0:256]
# A photo-like image: large smooth structures, a bright object, a little grain.
base = 110 + 55*np.sin(x/70) + 35*np.cos(y/55)
base += 70 * (((x-90)**2 + (y-70)**2) < 45**2)
base += 50 * (((x-180)**2 + (y-185)**2) < 30**2)
base = np.clip(base + rng.normal(0, 4, (256, 256)), 0, 255)
BASE = Image.fromarray(base.astype(np.uint8))

def phash(im, hash_size=8, factor=4):
    """Zauner's pHash: shrink, DCT, keep the low frequencies, compare to their median."""
    s = hash_size * factor
    a = np.asarray(im.convert("L").resize((s, s), Image.LANCZOS), float)
    d = dctn(a, norm="ortho")[:hash_size, :hash_size]
    flat = d.flatten()[1:]                       # drop the DC term: it is only average brightness
    return (flat > np.median(flat)).astype(np.uint8)

def hamming(a, b): return int((a != b).sum())

def jpeg(im, q):
    b = io.BytesIO(); im.save(b, "JPEG", quality=q); return Image.open(b)

variants = {
    "identical file":        BASE,
    "JPEG quality 90":       jpeg(BASE, 90),
    "JPEG quality 40":       jpeg(BASE, 40),
    "resized 50% and back":  BASE.resize((128,128), Image.LANCZOS).resize((256,256), Image.LANCZOS),
    "brightness +25":        Image.fromarray(np.clip(base+25, 0, 255).astype(np.uint8)),
    "cropped 10% and back":  BASE.crop((26,26,230,230)).resize((256,256), Image.LANCZOS),
    "horizontally flipped":  BASE.transpose(Image.FLIP_LEFT_RIGHT),
    "rotated 5 degrees":     BASE.rotate(5, resample=Image.BICUBIC),
    "a genuinely different image":
        Image.fromarray(np.clip(110 + 55*np.sin(y/70) + 35*np.cos(x/55)
                                + rng.normal(0,4,(256,256)), 0, 255).astype(np.uint8)),
}

h0 = phash(BASE)
sha = lambda im: hashlib.sha256(im.convert("L").tobytes()).hexdigest()[:12]
print("variant                       exact hash match   pHash distance (of 63)   near-dup at <=6?")
for name, im in variants.items():
    d = hamming(h0, phash(im))
    print(f"{name:28s} {str(sha(im) == sha(BASE)):>16s} {d:24d} {str(d <= 6):>18s}")

print("\nThe four rows that matter:")
print("  exact hashing catches only the first row. Re-saving a JPEG defeats it entirely.")
print("  pHash catches re-compression, resizing and brightness changes.")
print("  pHash MISSES flips, rotations and crops. Those need an embedding, not a hash.")
Output
variant                       exact hash match   pHash distance (of 63)   near-dup at <=6?
identical file                           True                        0               True
JPEG quality 90                         False                        0               True
JPEG quality 40                         False                        0               True
resized 50% and back                    False                        0               True
brightness +25                          False                        6               True
cropped 10% and back                    False                       24              False
horizontally flipped                    False                       32              False
rotated 5 degrees                       False                       10              False
a genuinely different image             False                       32              False

The four rows that matter:
  exact hashing catches only the first row. Re-saving a JPEG defeats it entirely.
  pHash catches re-compression, resizing and brightness changes.
  pHash MISSES flips, rotations and crops. Those need an embedding, not a hash.

Reading the output

Exact hashing catches one row out of nine. Re-saving the same photograph as JPEG at quality 90 produces a different byte hash. As far as the viewer is concerned, nothing changed. This is why deduplication by sha256 alone finds almost nothing on a web-scraped set.

pHash is unaffected by compression and resizing. Distance 0 for JPEG at both 90 and 40, and for a 50 percent resize round trip. Content delivery networks and messaging apps apply those transformations constantly. They are exactly what the design targets. The DCT keeps only the lowest frequencies, and compression and resampling mostly damage the high ones.

Brightness costs 6 bits. Right at a typical threshold. The DC term is dropped, which removes the mean. But a global brightness shift still perturbs the surviving coefficients.

Crop, flip and rotation are misses. 24, 32 and 10 respectively. The flipped image scores 32 out of 63, which is exactly what two unrelated images score. As far as pHash is concerned, a mirrored photograph is a different photograph.

That is a design consequence, not a bug. A hash designed to be invariant to arbitrary geometry would also collide on genuinely different images.

Choosing the threshold, with a real curve

Never take a threshold from a blog post. Build a small labelled set and measure.

threshold.py
import numpy as np, io, itertools
from scipy.fft import dctn
from PIL import Image
rng = np.random.default_rng(1)
y, x = np.mgrid[0:128, 0:128]

def scene(seed):
    r = np.random.default_rng(seed)
    a = 110 + 55*np.sin(x/(30+r.integers(0,40))) + 35*np.cos(y/(25+r.integers(0,40)))
    cx, cy = r.integers(20, 108, 2)
    a = a + 70 * (((x-cx)**2 + (y-cy)**2) < 30**2)
    return Image.fromarray(np.clip(a + r.normal(0, 4, a.shape), 0, 255).astype(np.uint8))

def phash(im, n=8, f=4):
    a = np.asarray(im.convert("L").resize((n*f, n*f), Image.LANCZOS), float)
    d = dctn(a, norm="ortho")[:n, :n].flatten()[1:]
    return (d > np.median(d)).astype(np.uint8)

def jpeg(im, q):
    b = io.BytesIO(); im.save(b, "JPEG", quality=q); return Image.open(b)

# 30 distinct photos, plus 10 near-duplicates of the first ten.
imgs = [scene(s) for s in range(30)]
truth = []                                     # (i, j) pairs that really are the same photo
for k in range(10):
    variant = [jpeg(imgs[k], 45), imgs[k].resize((90, 90)).resize((128, 128)),
               Image.fromarray(np.clip(np.asarray(imgs[k], float)+20, 0, 255).astype(np.uint8))][k % 3]
    truth.append((k, len(imgs))); imgs.append(variant)

H = [phash(im) for im in imgs]
pairs = list(itertools.combinations(range(len(imgs)), 2))
real = set(truth)
print("threshold   flagged   correct   precision   recall")
for t in range(0, 17, 2):
    flagged = [p for p in pairs if int((H[p[0]] != H[p[1]]).sum()) <= t]
    tp = sum(p in real for p in flagged)
    prec = tp / len(flagged) if flagged else 1.0
    print(f"{t:9d} {len(flagged):9d} {tp:9d} {prec:11.1%} {tp/len(real):8.1%}")
print(f"\n{len(pairs)} pairs compared, {len(real)} of them are true duplicates.")
print("Pick the threshold from a table like this one, on YOUR images. Never from a blog post.")
Output
threshold   flagged   correct   precision   recall
        0         4         4      100.0%    40.0%
        2         8         8      100.0%    80.0%
        4        10        10      100.0%   100.0%
        6        10        10      100.0%   100.0%
        8        12        10       83.3%   100.0%
       10        15        10       66.7%   100.0%
       12        20        10       50.0%   100.0%
       14        22        10       45.5%   100.0%
       16        27        10       37.0%   100.0%

780 pairs compared, 10 of them are true duplicates.
Pick the threshold from a table like this one, on YOUR images. Never from a blog post.

Thresholds 4 and 6 are perfect on this set. Below that, recall falls. Above 6, precision decays steadily while recall has nothing left to gain. On real photographs the plateau is narrower and the numbers will differ; the shape of the curve will not.

Build this table on 50 known pairs from your own data. It takes half an hour and it is the difference between a defensible threshold and a guess.

Scaling past a few thousand images

The script above compares every pair, which is quadratic. At a million images that is 500 billion comparisons.

The standard fix is locality-sensitive hashing by banding. Split the 64-bit hash into 8 bands of 8 bits. Index each band in a dictionary. Compare only pairs sharing at least one band exactly. By the pigeonhole principle, two hashes within a Hamming distance of 6 must agree on at least one band. So this loses nothing at that threshold, and it shrinks the candidate set enormously.

For embedding-based deduplication, use an approximate nearest-neighbour index. FAISS is the usual choice, and the mechanics are the same as those covered in vector databases.

When you need embeddings instead

Crops, flips, rotations, colour grading and heavy overlays defeat pHash. For those, compare learned features rather than pixel patterns.

Copy-detection embeddings are the right tool. They are descriptors trained for one question: is this a modified copy of that. Published deduplication pipelines compute such descriptors and retrieve the nearest neighbours for each image. Pairs above a cosine-similarity threshold near 0.75 are then collapsed. A general-purpose model such as CLIP also works, and is more available. It will match semantically similar but genuinely distinct images, so set its threshold more conservatively.

The practical pipeline is a cascade. Byte hash first, then pHash for the cheap majority, then embeddings on what remains.

Common mistakes

Deduplicating after splitting. Always before. The next lesson is entirely about what happens when you do not.

Deduplicating with sha256 alone and declaring the set clean. See the first table.

Deleting duplicates without recording them. Keep the group. Which images were collapsed, and which representative was kept, is information you will want later. Once deleted it is gone.

Ignoring near-duplicates within the training set. They over-weight examples silently. Keeping one per group and recording the group size lets you decide later whether to reweight.

Using a single threshold across a mixed dataset. Screenshots, scanned documents and natural photographs have very different hash distributions. Set thresholds per source type.

Try it yourself

Add a dhash variant, which compares each pixel with its right-hand neighbour in a small resized image. Run the same threshold sweep. Compare the two curves. dhash is faster and typically more sensitive to fine structure. Knowing which one wins on your data is a twenty-minute experiment.

What to learn next

Researcher — Mathematics and papers.

What a perceptual hash is doing

A perceptual hash maps an image to a short binary code. Perceptually similar images then have a small Hamming distance. It is a locality-sensitive hash for a perceptual metric. The requirements are the reverse of a cryptographic hash. Collisions on similar inputs are wanted, and sensitivity to small changes is a defect.

The pHash construction (Zauner, 2010) is the standard. Convert to greyscale, resize to $32 \times 32$, and take the 2D DCT. Keep the top-left $8 \times 8$ low-frequency block, and discard the DC coefficient. Threshold the remaining 63 values at their median. Median thresholding is what gives brightness and contrast invariance. A monotonic intensity change preserves the ordering of coefficients relative to their median.

The invariances follow from the construction:

TransformationSurvivesWhy
JPEG re-compressionYesDamages high frequencies, which are discarded
ResizeYesThe image is resized to a fixed grid first
Brightness, contrastMostlyDC discarded, median threshold is scale-invariant
Small rotationPartiallyLow frequencies shift gradually
CropNoChanges the spatial support, so the DCT basis alignment moves
FlipNoNothing in the pipeline is reflection-invariant

The published analysis of perceptual hashing for near-duplicate detection makes the same point. DCT-based pHash detects near-exact copies differing only in compression or scaling. It cannot capture geometric transforms such as flips, crops or colour shifts. So it is unreliable as a general near-duplicate detector.

Adversarial robustness

Perceptual hashes are not adversarially robust, and this matters wherever they are used for enforcement rather than for hygiene. Jain et al. (2021), Adversarial Detection Avoidance Attacks, attack perceptual-hashing-based client-side scanning. Visually imperceptible modifications evade detection. The converse also works: an innocuous image can be built to collide with a target hash.

For dataset deduplication this is not a threat model, since nobody is attacking your training set. It matters for content moderation and provenance soft bindings. The C2PA specification lists watermarks and fingerprints as soft binding mechanisms, precisely because they are not statistically unique.

Embedding-based copy detection

The successor approach learns descriptors for the copy-detection task directly. Self-supervised training and aggressive augmentation cover exactly the transformations hashes fail on.

SSCD (Pizzi et al., 2022) adapts self-supervised contrastive learning to copy detection. An entropy regularisation term spreads descriptors across the embedding space, improving retrieval separation. Large-scale text-to-image dataset pipelines use it in a documented pattern. Compute 512-dimensional SSCD embeddings, and retrieve $k=64$ nearest neighbours per image. Collapse pairs whose cosine similarity exceeds 0.75.

The trade against pHash is explicit. Embeddings cost a forward pass per image, plus an approximate nearest-neighbour index rather than a dictionary lookup. They handle crops, flips, rotations and overlays. The standard production design is a cascade: exact hash, then pHash, then embeddings on the survivors.

Why this changes benchmark conclusions

Barz and Denzler (2020), Do We Train on Test Data? Purging CIFAR of Near-Duplicates, measured the overlap. 3.3 percent of the CIFAR-10 test set has duplicates in the training set. For CIFAR-100 the figure is 10 percent. They constructed ciFAIR by replacing every duplicate test image with a new image from the same domain. On re-evaluation they reported accuracy drops of 9 to 14 percent, relative to the original figures.

Read that carefully. A tenth of CIFAR-100's test set was memorisable. Removing that shortcut moves reported accuracy by a double-digit relative amount. Every CIFAR result published before this correction includes some memorisation.

The same audit has been run elsewhere and finds the same thing. Face recognition datasets carry substantial duplication (Boutros et al., 2024). Every large web-scraped corpus does too, by construction, because the web itself is heavily duplicated.

The design rule

Deduplicate before splitting, at the group level, and record the groups.

Deduplicating after splitting cannot repair leakage that has already happened. Removing duplicates within a split without checking across splits misses the case that matters. And a group record maps each retained image to the set it represents. That lets the reweighting decision be revisited later. It also lets a leakage audit be re-run against a new split.

Report the deduplication method, the threshold, and the number of groups collapsed as part of the dataset description. A dataset paper that does not state its deduplication procedure has not established that its test set measures generalisation.

References

What to learn next