Duplicate Line Remover
Remove duplicate lines from a list, with case and whitespace options.
What is Dedupe Lines?
The Duplicate Line Remover cleans pasted lists (emails, tags, URLs, log lines) by removing duplicate lines in a single pass, with configurable comparison (case sensitivity, whitespace trimming) and an explicit removed-lines report for trust.
How it works
A Set-based single pass keeps the first occurrence of each line. When case-insensitive or trim modes are on, lines are compared by their normalized form. The removed lines are collected and shown alongside the cleaned result.
- Paste your list. One item per line — emails, names, URLs, log output, anything list-shaped.
- Configure comparison. Toggle case sensitivity, whitespace trimming, blank-line filtering and sorting to match how strict your dedupe needs to be.
- Read the report. The cleaned result and the removed lines (with count) are shown side by side, so you can verify before trusting the output.
Examples
Clean a mailing list
A pasted list with repeated email addresses is deduplicated keeping the first occurrence of each.
Case-insensitive dedupe
With case-insensitivity on, 'alice@x.com' and 'Alice@x.com' count as duplicates; with it off they are both kept.
Common mistakes
Whitespace makes lines look different
'foo' and 'foo ' are distinct lines. Enable 'Trim whitespace' to treat them as duplicates.
Case sensitivity default
Comparison is case-sensitive by default — flip the toggle if your data varies in capitalization.
Alternatives
CSV Deduplicator
For deduplicating rows in structured CSV data by key columns.
Text Sorter
Often used after dedupe — sort the cleaned list alphabetically.
Frequently asked questions
Why does removing duplicates change my file's line order?
It doesn't by default — first occurrences keep their original position. Only enabling 'Sort result' reorders the output.
Does it handle case-insensitive duplicates?
Yes. Toggle 'Case-sensitive' off to compare lines ignoring capitalization — the first occurrence is still the one kept.