URL normalizer: find duplicate URL variants

Many strings can point at one page. Capital letters in the host, an explicit port, a dot segment in the path, a tracking parameter or a trailing slash each make a new URL while the content stays the same. This URL normalizer finds those groups in a list you paste, in your browser, without sending the list anywhere.

What this URL normalizer checks#

The tool applies two kinds of normalization, and keeps them apart.

RFC 3986 rules, always on. Section 6.2 of RFC 3986 describes rewrites that produce an equivalent URI:

• lowercase the scheme and host
• write percent-encoding hex in uppercase, such as %3A
• decode percent-encoded unreserved characters, so %7E becomes ~
• remove . and .. dot segments from the path
• drop a port that equals the scheme’s default, such as :80 on http
• treat an empty path as /

The path keeps its case. RFC 3986 treats it as case-sensitive, and Google treats /APPLE and /apple as distinct URLs.

Site policy, your choice. Six checkboxes, all on by default: prefer https, drop www., drop the trailing slash, drop tracking parameters, sort parameters, and drop the #fragment. None of these is an RFC rule. Whether www and the bare domain are one site is a decision about your site, so untick anything that does not match how it works.

How to read the results#

The summary line reads “N URLs collapse to M distinct”. The gap between the two is the size of the duplication. Next comes the number of groups that would need one canonical, and a count of any lines that were not http or https URLs.

Each group shows a canonical candidate, the normalized form, with the original URLs beneath it. Only groups with two or more different originals are listed. A URL pasted twice is not a variant of itself.

Google names redirects and rel="canonical" as strong signals for the canonical URL and a sitemap as a weak one, and says combining methods raises the chance your choice is used. The practical fix for each group is a redirect from every variant, plus a self-referencing canonical on the page. Google also prefers HTTPS over an equivalent HTTP page as canonical, unless there are issues such as an invalid certificate.

Worked example#

The tool loads with eight URLs. With every option on, the summary reads “8 URLs collapse to 5 distinct. 3 groups of variants that would need one canonical.”

Canonical candidate	Variants grouped under it
https://example.com/shoes	https://Example.com/shoes/, and the same path on plain http with :80
https://example.com/shoes?color=red	https://www.example.com/shoes?utm_source=news&color=red, https://example.com/shoes?color=red
https://example.com/about	https://example.com/about#team, https://example.com/about

The other two lines normalize but have no partner. https://example.com/a/./b/../shoes becomes /a/shoes, and %7Euser becomes ~user. Untick “Drop www.” and the second group splits in two, leaving 6 distinct URLs and 2 groups.

Common mistakes it catches#

• http and https both live. Google lists protocol variants among the causes of duplicate URLs.
• www and non-www both served. Also named in Google’s guide as a duplicate cause.
• Tracking parameters in internal links. Google uses gclid as its example of a parameter that creates a duplicate URL. A tagged internal link creates a new address for the same page.
• Session IDs in URLs. Google’s URL structure guide lists session IDs among the causes of too many URLs and suggests cookies instead.
• Parameters in a different order. ?a=1&b=2 and ?b=2&a=1 are different strings. Sorting shows that they are the same page.

Related tools and reading#

• Canonical tag checker to confirm the canonical each page declares
• Faceted navigation parameter analyzer for parameter-heavy URLs
• Site migration redirect map builder to turn groups into redirects
• SearchPilot test: canonical alone matched a redirect for duplicates