How does duplicate detection work?

Each line is compared to all previously seen lines. The first occurrence is kept and any subsequent identical lines are dropped. The relative order of unique lines is preserved by default.

What does case-insensitive matching do?

'Hello' and 'hello' would normally be treated as different lines. With case-insensitive matching enabled, they are considered duplicates and only the first occurrence is kept.

What does trim whitespace do?

Leading and trailing spaces are stripped from each line before comparison. This means ' hello ' and 'hello' are treated as the same line.

Does sorting affect which duplicate is kept?

Sorting reorders the final output alphabetically but does not affect which line is kept — duplicates are still removed in the original top-to-bottom order before sorting.

Private by defaultRuns in your browser

Duplicate line remover

Paste your text and get back only unique lines. Useful for cleaning up lists, log output, CSV data, and any text with repeated entries.

InputOne item per line

Unique lines

Case-sensitiveTrim whitespaceSort output

Quick samplesUseful for testing

Duplicate Remover

About this tool

Deduplicating a list is one of those tasks that is trivial in principle and fiddly in practice, because the definition of duplicate is rarely as simple as byte-for-byte identical. Two entries differing only by a trailing space, by capitalisation, or by an invisible carriage return are duplicates to a human and distinct strings to a computer. That gap is where most of the work lies when cleaning up email lists, log samples, SQL result sets, exported CSV columns, keyword lists, or IP addresses collected from several sources. Order matters too. Keeping the first occurrence preserves the original sequence, which is what you want when the list is ranked or chronological; keeping the last is what you want when later entries supersede earlier ones, as in a change log. Sorting after deduplication makes the result easier to scan and diff but destroys any meaningful original ordering. This tool removes duplicate lines with control over case sensitivity, whitespace trimming, which occurrence to keep, and whether to sort the result. Everything runs in your browser, so pasted data stays on your machine.

No signup requiredRuns in your browserInstant results

How to use

1
Paste your list or text with duplicate lines into the input field.
2
Choose whether to keep the first or last occurrence of each duplicate.
3
Optionally sort the result after deduplication.
4
Click Copy to copy the cleaned list.

Why use this tool?

→
Clean up exported lists or CSVs that contain repeated rows before importing.
→
Deduplicate log output or error messages to focus on distinct events.
→
Remove repeated entries from a manually assembled list of IDs, emails, or URLs.

ExamplesInput → output

Remove duplicate lines

Input

apple
banana
apple
orange
banana

Output

apple
banana
orange

Deduplicate a comma list

Inputa, b, a, c, b, d

Outputa, b, c, d

Common errorsAnd how to fix them

Visually identical lines are not treated as duplicates

Cause: Invisible differences. Trailing spaces, tabs versus spaces, a stray carriage return from a Windows-formatted file, or a non-breaking space pasted from a web page all make two lines distinct.

Fix: Enable whitespace trimming, which resolves most cases. Non-breaking spaces and zero-width characters survive trimming and need a search-and-replace first.

Case-insensitive mode removed entries that were genuinely different

Cause: Case is significant in some data and not others. Linux file paths, base64 strings, and many API keys are case-sensitive; email local parts are case-sensitive by specification even though most providers ignore it.

Fix: Use case-insensitive matching only for data where case carries no meaning. When in doubt, deduplicate case-sensitively and review the near-duplicates by hand.

The output order is not what was expected

Cause: Either sorting was left enabled, or the keep-first and keep-last settings interact with ordering in a way that is easy to misread — keeping the last occurrence moves each surviving entry to the position of its final appearance.

Fix: Turn off sorting to preserve original order, and choose keep-first when the sequence is meaningful. Keep-last is for supersession, where the newest entry should win.

Frequently asked questionsCommon questions answered

These answers explain common duplicate remover tasks, expected input formats, and edge cases so both visitors and search engines can understand what this tool does.

How does duplicate detection work?: Each line is compared to all previously seen lines. The first occurrence is kept and any subsequent identical lines are dropped. The relative order of unique lines is preserved by default.
What does case-insensitive matching do?: 'Hello' and 'hello' would normally be treated as different lines. With case-insensitive matching enabled, they are considered duplicates and only the first occurrence is kept.
What does trim whitespace do?: Leading and trailing spaces are stripped from each line before comparison. This means ' hello ' and 'hello' are treated as the same line.
Does sorting affect which duplicate is kept?: Sorting reorders the final output alphabetically but does not affect which line is kept — duplicates are still removed in the original top-to-bottom order before sorting.