Remove Duplicate Lines
De-duplicate a list without reordering it. Keep the first occurrence or the last, decide whether the comparison cares about case, and decide whether leading spaces are part of the line or just noise from a copy and paste. Furtu reports how many lines went and how many unique ones remain.
Last is what you want when a later line overrides an earlier one, which is how a settings file behaves.
Off, Apple and apple are the same line.
Off, an indented value and the same value unindented are different lines.
How remove duplicate lines works
- Paste the listOne entry per line, in any order.
- Set the comparisonCase, surrounding whitespace, and which copy to keep.
- Check the countsLines in, lines out and how many were removed.
What you get
Drop repeated lines, keeping either the first or the last copy. Everything happens inside this page: the file is read by your browser, transformed in memory and handed straight back to you as a download. There is no upload queue, no waiting for a server, and nothing left behind when you close the tab.
Supported formats
This tool works on text you paste or type, so there is no file format to worry about. Nothing you type is sent anywhere.
Limitations, stated up front
- Lines are compared whole. Two entries that differ only in internal spacing are different lines.
- The final newline is preserved when the input had one, so a de-duplicated file still ends the way the original did.
Frequently asked questions
What counts as the same line?
With the default settings, a line matches another that differs only in case or in leading and trailing spaces. Turn the toggles off for an exact comparison. Neither is wrong; they answer different questions, and which one is right depends on whether the whitespace is data.
Does it sort the result?
No. Order is preserved, which is usually what you want from a de-duplicator and is what makes the sort-lines tool a separate step. Use it in sequence if you want both.
What about empty lines?
They are kept, and they count as lines. A file with forty blank lines will report one unique blank line rather than dropping them, because silently removing lines is a surprise in a tool whose job is removing specific lines.
How large can the input be?
A set of keys is held in memory for the comparison, so a few hundred thousand lines is comfortable and the result is exact. Beyond that the browser, not Furtu, runs out of room.
Are the lines in my list uploaded?
No. De-duplication runs in your browser, so a list of email addresses or customer IDs stays on your device.