What is URL normalization?

URL normalization converts URLs into a canonical form so that different addresses for the same page become one entry. Without it, a crawler treats every variation as a new page and fetches the same content repeatedly.

The scale of the problem is easy to underestimate. These are all the same page:

https://example.com/products
https://example.com/products/
https://example.com/products?utm_source=twitter
https://example.com/products?ref=nav&utm_campaign=spring
https://example.com/products#reviews
https://EXAMPLE.com/products
http://example.com/products

Seven URLs, one page. On a real site with tracking parameters, sort orders and filters, one page can generate hundreds.

What normalization does

• Lowercase the scheme and host. Domains are case-insensitive; paths are not, so leave the path alone.
• Drop the fragment. Everything after # is a client-side anchor and never reaches the server.
• Remove the default port. :80 on HTTP and :443 on HTTPS add nothing.
• Resolve dot segments. /a/b/../c becomes /a/c.
• Decide about the trailing slash. Pick a rule and apply it consistently. Which rule matters less than consistency.
• Sort query parameters. ?b=2&a=1 and ?a=1&b=2 are the same request to nearly every server.
• Strip parameters that do not change content. The utm_* family, fbclid, gclid, ref and session identifiers.
• Normalise percent-encoding, so %7E and ~ do not diverge.

The part that needs judgement

Which query parameters to strip is site-specific and cannot be fully generalised. ?page=2 changes the content and must stay. ?utm_source=x does not and should go. ?sort=price is genuinely ambiguous: same products, different order, and whether that counts as a different page depends on what you are collecting.

Stripping too aggressively loses pages. Stripping too little wastes crawl budget. A workable default is an allowlist of parameters known to matter for a given site, with everything else dropped, since new tracking parameters appear constantly and known-good ones rarely change.

The signal the site gives you

Many pages include a <link rel="canonical"> tag naming the URL the site considers authoritative. That is the site telling you exactly what you are trying to work out, and honouring it is more reliable than any rule you write. Read it, and prefer it over your own normalisation where the two disagree.

Why it is worth doing early

Normalization has to happen before a URL enters the frontier and before the seen-set check, because its whole purpose is making that check work. Adding it later means a seen-set already full of duplicates, and the usual fix is starting the crawl again.