Format Code Now
/
TextRemove Duplicate Lines Next: Sort Lines ›

Remove Duplicate Lines

TX-08

Paste a list and get it back with every repeated line removed. The first copy of each line stays, in its original place, so the order does not change.

Runs in your browser. Nothing is uploaded.

Lines

Unique lines

How to use it

  1. Paste your lines into the input box, or open a text file and drop it on the box.
  2. Leave the options off for an exact match, or tick Ignore upper and lower case and Ignore spaces at the start and end.
  3. Press Remove duplicates. The list updates as you type.
  4. Press Show duplicates to see each repeated line once, with how many times it appears.
  5. Copy the result or download it as a .txt file.

How lines are compared

By default two lines are duplicates only when every character matches, including case, spaces and tabs. So Apple and apple are two lines, and so are "banana" and "banana " with a trailing space. The two comparison options relax that. The line that is kept is written exactly as it first appeared, even when it matched a later line only after case or spaces were ignored. Empty lines are lines too: the second and later empty lines are removed unless you tick Keep every empty line.

Finding which lines repeat

Show duplicates answers the question "what is repeated, and how often?" instead of cleaning the list. Each repeated line is listed once, in the order it first appears, with its count, a tab, and the text. Lines that appear only once are left out. If nothing repeats, you get the message No duplicate lines found. The same options apply, so you can check for repeats that differ only in case or in stray spaces before you remove them.

What it does not do

It compares whole lines only. It does not spot near matches such as a typo, a plural, or the same address written two ways, and it does not sort the list. To put the unique lines in order, run the result through the Sort Lines tool. Line endings are kept: if your text uses Windows line breaks (CRLF) the output does too, and a final line break at the end of the text stays.

Questions people ask

How do I remove duplicate lines from a list?

Paste the list into the input box and press Remove duplicates. Each line is kept the first time it appears and dropped after that. The remaining lines stay in their original order, so nothing is shuffled or sorted. Copy the result or download it as a text file.

Does it keep the first or the last duplicate?

It keeps the first one. Later copies are removed, and the surviving lines keep their original order. If you need the last copy of each line instead, reverse the line order first with the Reverse Text tool, remove the duplicates, then reverse the result back.

How do I remove duplicates that differ only in capital letters?

Tick Ignore upper and lower case. Then Apple, apple and APPLE count as the same line and only the first one is kept, written as you typed it. Tick Ignore spaces at the start and end as well to treat a line with a trailing space as a match.

How can I see which lines are duplicated before deleting them?

Press Show duplicates. It lists each repeated line once with the number of times it appears, in the order each line first showed up. Lines that appear only once are not listed. This is a quick way to check a list before you clean it.

Does it remove duplicate empty lines too?

Yes by default: an empty line is a line, so the second and later empty lines are removed. Tick Keep every empty line if blank lines are separators you want to preserve. If you only want to get rid of blank lines, the Remove Empty Lines tool is the better fit.

Can it handle a large list?

It handles lists of many thousands of lines in the browser without trouble, since it only compares each line against the lines it has already seen. Very large files are limited by your device memory, and the file open box has its own size limit.

About this tool

Keeps the first occurrence of each line and drops later copies, so the order of what remains does not change. By default two lines must match exactly: case, spaces and tabs all count, so Apple and apple are different lines. The first two options change only how lines are compared; the line that is kept is written exactly as it first appeared. Empty lines count as lines too, so a second empty line is removed unless you tick Keep every empty line. Show duplicates lists each repeated line once with how many times it appears (the count, a tab, then the first copy), in the order they first appear. Mixed line endings are written back out with Windows endings (CRLF) if the text had any, otherwise with LF. Lines are compared as whole lines; nothing is compared by meaning or spelling.

Format Code Now: free, no ads, no accounts. Your input stays in your browser.