What is deduplication in web crawling?
Deduplication is the process of detecting and removing repeated records. In web crawling, it means each page is fetched and stored once, even though the crawler meets it many times: linked from every listing page, reachable through several URL parameters, or repeated as a sponsored item on every page of a category.
Without it, a crawler wastes requests on content it already has, returns inflated item counts, and can loop forever. With it, the output is a clean list where every row is a distinct page.
Three levels of deduplication
| Level | What counts as a duplicate | How it is detected |
|---|---|---|
| URL | The same address, character for character | A seen-set of URLs |
| Normalized URL | Different spellings of one address | URL normalization, then a seen-set |
| Content | Different URLs serving the same page | Canonical URL, or a hash of the page content |
Exact URL matching catches the least. Normalization handles tracking parameters, trailing slashes, letter case in the host, fragments and parameter order. Content-level checks catch the rest, such as one product listed under two category paths.
How a crawler deduplicates
- Normalize the URL before anything else.
- Check the seen-set. If the normalized URL is there, skip it. If not, add it and enqueue it. Checking at enqueue time, rather than at fetch time, keeps the queue small.
- After fetching, read the canonical and, where it is trustworthy, key the stored record on it.
- Optionally hash the content to catch near-identical pages served at unrelated URLs.
At scale, the seen-set itself becomes an engineering problem. Millions of URLs fit in memory as a set of hashes; billions call for a database, a key-value store or a probabilistic structure such as a Bloom filter, which uses little memory at the cost of rare false positives.
Exact vs near-duplicate detection
Exact hashing (MD5, SHA-1) only matches byte-identical content, and a page with a changing timestamp or ad slot never hashes the same twice. Near-duplicate techniques such as SimHash and MinHash compare fingerprints of the text, so two pages that differ only in boilerplate still match. Search engines use them heavily; most data projects get far with normalization plus canonical URLs.
Common mistakes
- Deduplicating too late, after fetching, so the duplicate requests have already been spent.
- Over-aggressive normalization that strips a parameter which does change content, such as
pageorvariant, and silently drops real pages. - Keeping the seen-set only in memory, so a restarted crawl forgets everything and refetches.
- Counting listing items without deduplication, which inflates totals with sponsored items that repeat on every page.
Data deduplication outside crawling
The term also has a storage meaning: removing duplicate blocks of data so backups and file systems store each block once. The idea is the same, identify repeats and keep one copy, applied to bytes on disk instead of web pages.
How Zenrows Crawl handles this
Deduplication is built into Zenrows Crawl. Every link on a page it opens is checked against your include and exclude patterns and against the links already found, so each URL appears in the results once. Each crawl reports duplicates removed alongside pages fetched and URLs found, so you can see how much repetition the site had. The fields are listed in the Crawl endpoint reference.
Frequently asked questions
What is URL deduplication?
Making sure each URL is processed once. It combines URL normalization, so different spellings collapse, with a record of URLs already seen.
How do you detect duplicate web pages?
Compare normalized URLs first, then canonical tags, then content hashes. Near-duplicate methods like SimHash catch pages that differ only in small details.
What is a Bloom filter in web crawling?
A compact data structure that answers "have I seen this URL?" using very little memory. It never misses a URL it has seen, but occasionally reports one as seen when it was not.
Why does my scraper return more items than the site shows?
Usually duplicates: sponsored or pinned items repeated on every page, or the same item reached through several URLs. Deduplicate on a normalized item URL.
Go deeper on the blog
In the docs
Last updated: Oct 9, 2026