How perceptual hashing spots a duplicate

Open a photo gallery, a product page, or a news article and the same image is often on the page several times: a 200-pixel thumbnail, an 800-pixel preview, a 2,000-pixel full-size, maybe a slightly re-compressed copy served by a CDN. To a human they're obviously the same picture. To a computer, they're four completely different files.

Bulk Image Downloader's "Duplicates" view groups those copies so you download one, not five. The trick that makes it possible is perceptual hashing — and it's simpler than it sounds.

Why exact hashing doesn't work

The usual way to check if two files are identical is a cryptographic hash like MD5 or SHA-256. Feed in the bytes, get out a fixed-length fingerprint. If two files have the same hash, they're the same file.

That's precisely the wrong tool here. Cryptographic hashes are designed so that changing a single byte produces a completely different result. Resize an image by one pixel, save it at 90% quality instead of 92%, or let a CDN re-encode it, and the exact hash changes entirely — even though the picture looks the same. Exact hashing answers "are these the same bytes?" We need to answer a different question:

Do these two images look the same?

The idea: a fingerprint of what the image looks like

A perceptual hash produces a fingerprint from the content of an image rather than its bytes. The goal is the opposite of a cryptographic hash: images that look similar should get similar fingerprints, and small changes should cause small differences.

Bulk Image Downloader uses a well-known and pleasantly cheap variant called dHash (difference hash). Here's the whole recipe.

1. Shrink it. Scale the image down to a tiny 9×8 grid and convert to greyscale. All the fine detail — the exact pixels, the resolution, the file size — is thrown away. What survives is the rough structure of light and dark.
2. Compare neighbours. For each row, walk left to right and ask a single yes/no question about each pixel: is it brighter than the pixel immediately to its right? Nine pixels per row gives eight comparisons, and eight rows gives 8 × 8 = 64 yes/no answers.
3. Read off the bits. Write each "yes" as a 1 and each "no" as a 0. That 64-bit number is the image's fingerprint.

Because the fingerprint is built from relative brightness between neighbouring pixels, it barely moves when you resize the image, nudge the compression, or shift the overall brightness. The gradient from light to dark across a mountain ridge is the same whether the photo is 200 pixels wide or 2,000. That's exactly the robustness we want.

Comparing two fingerprints

Once every image has a 64-bit fingerprint, comparing them is trivial: line up two fingerprints and count how many bits differ. That count is the Hamming distance.

A: 10110100 11100101 01101100 …
B: 10110100 11000101 01101100 …
                ↑
   differ in 1 place  →  Hamming distance = 1

• Distance 0 — the two images have the identical structure. Almost always the same picture at different sizes.
• A small distance — a near-duplicate. A re-compression, a light edit, a watermark in the corner.
• A large distance — genuinely different images.

Bulk Image Downloader groups any images whose fingerprints are within a distance you control. The Match sensitivity slider is exactly this threshold: at the strict end it only groups near-identical copies; loosen it and it starts pulling in more distant look-alikes. The default sits at a distance of about 6 out of 64 — tight enough to avoid false matches, loose enough to catch the everyday resize-and-recompress copies.

What it catches — and what it doesn't

dHash is deliberately a blunt instrument, and that's a feature. It reliably catches the cases that actually clutter a download folder: the same photo at five sizes, a JPEG re-saved by a CDN, a logo with and without a small badge.

It intentionally does not treat a hard crop, a horizontal flip, or a heavy colour grade as the same image — because, for someone downloading pictures, those usually are different pictures. If you want more or fewer matches, that's what the sensitivity slider is for; you stay in control of the final call.

Why this approach

There are fancier perceptual hashes — pHash uses a discrete cosine transform, and neural embeddings can match across crops and rotations. But they're heavier, slower, and for a browser extension that has to fingerprint hundreds of images the moment you click, dHash hits a sweet spot: a few milliseconds per image, no libraries to ship, no data to upload, and results you can reason about. Sometimes the small, legible tool is the right one.