What is a canonical URL?
A canonical URL is the address a website declares as the main version of a page that can be reached at several URLs. The site states it with a <link rel="canonical"> tag in the page's <head>, and search engines use it to group the duplicates, rank one version, and show that one in results. For crawlers and scrapers, it is the site telling you which of the many addresses you met is the real one.
The same product often lives at /product/blue-hoodie, /product/blue-hoodie?color=blue, /men/hoodies/blue-hoodie and /product/blue-hoodie?utm_source=newsletter. The canonical tag on all four points to one of them.
How to declare a canonical URL
<link rel="canonical" href="https://example.com/product/blue-hoodie" />
| Method | Where | Best for |
|---|---|---|
rel="canonical" link tag |
HTML <head> |
Most web pages |
Link HTTP header |
Response headers | PDFs and other non-HTML files |
| 301 redirect | Server | Duplicates that should not exist at all |
| Sitemap inclusion | XML sitemap | A weak hint that a URL is preferred |
A canonical tag is a hint rather than a command. Google can pick a different canonical if other signals, such as internal links and redirects, disagree with the tag.
Canonical URL vs URL normalization
URL normalization is something your crawler does: rewriting URLs by rule, such as lowercasing the host, dropping fragments and stripping tracking parameters, so obvious duplicates collapse. A canonical URL is something the site tells you. Rules catch the generic cases; the canonical tag catches site-specific ones that no rule could guess, such as the same product listed under two categories.
A good crawler does both. Normalize every URL before it enters the queue, then record each fetched page's canonical and use it as the key for deduplication.
Common canonical mistakes
- Every page canonicalised to the homepage, usually a template bug, which tells search engines the whole site is one page.
- Paginated pages canonicalised to page one. Page 2 of a listing holds different items, so it should be self-canonical.
- Canonical pointing at a URL that redirects or returns 404.
- Relative URLs in the tag, which resolve unpredictably. Use absolute URLs.
- Conflicting signals, such as a canonical to URL A while the sitemap lists URL B.
Using canonical URLs when scraping
- Read the canonical, but verify it. Templates get it wrong. If the canonical's host or path looks unrelated to the page, fall back to your normalized URL.
- Store both URLs. Keep the URL you fetched and the canonical, which makes later debugging much easier.
- Do not follow it blindly across domains. A cross-domain canonical is legitimate for syndicated content, but it can lead a crawler outside its scope.
How Zenrows Crawl handles duplicates
Zenrows Crawl drops duplicate links as it crawls, so each URL appears in the results once, and the crawl reports how many duplicates it removed. The endpoint reference describes the result fields.
Frequently asked questions
What is a self-referencing canonical?
A canonical tag on a page that points to that page's own URL. It is good practice, because it tells search engines which version is preferred when parameters or tracking tags create copies.
Does a canonical tag stop a page from being crawled?
No. Crawlers still fetch the page; the canonical only says which URL should represent the group. To stop crawling, use robots.txt.
What is the difference between a canonical tag and a 301 redirect?
A 301 sends users and crawlers to another URL, so the duplicate is no longer reachable. A canonical leaves both URLs working and only signals which one is primary.
How do I find a page's canonical URL?
View the page source and search for rel="canonical", or check the Link response header. In code, parse the <head> and read the tag's href.
Go deeper on the blog
In the docs
Last updated: Oct 9, 2026