Free online tool

Free Online Remove Duplicate Lines

Paste a list and remove repeated lines in one click, with options for case, spaces and which copy to keep.

Remove Duplicate LinesFree online tool. Runs in your browser.
Live

Lists get messy. Email addresses collected from three forms, keywords pasted from two tools, names typed twice by a tired volunteer. Duplicates waste time and cause real problems, such as sending the same message twice. This tool removes repeated lines in a second and tells you which lines were repeated.

How duplicates are found

The tool reads your text one line at a time. For each line it makes a “key”: the line, after any clean-up you asked for. The first time it sees a key, it keeps the line. If it sees the same key again, it drops the line and adds one to a counter for that key. Because the keys are stored in a lookup table rather than compared one against another, the work grows in a straight line with the size of your list, so a list of tens of thousands of lines is still quick.

The clean-up options only change the key unless you say otherwise. For example, “ignore upper and lower case” treats Apple and apple as the same, but the line that is kept keeps its original letters.

What the options do

Option Effect Example
Ignore case Apple and apple match Keeps the first one
Trim spaces Ignores spaces at the ends “cat ” matches “cat”
Treat inner spaces as one Ignores doubled spaces “a b” matches “a b”
Remove empty lines Drops blank lines Tidy output
Collapse empty lines Keeps only one blank line in a row Keeps paragraph breaks
Keep the last Keeps the final copy instead of the first Latest version wins

By default the tool trims spaces, because a stray space at the end of a line is the most common reason that two “identical” lines are treated as different. It does not remove empty lines unless you ask, because those blank lines are often there on purpose.

How to use it

  1. Paste your list into the Input box, one item per line.
  2. Tick the options that match your data.
  3. Check the Result box. The tiles show how many lines went in, how many were kept and how many were removed.
  4. Open “Lines that repeated” to see which ones appeared more than once and how often.
  5. Copy the result or download it as a text file. Use “Use result as input” if you want to run a second clean-up pass.

A worked example

Try the example button. The input has nine lines: apple, Banana, apple, cherry, “banana ” with a trailing space, two blank lines, cherry and date. With the default settings, the second apple is a duplicate, the second cherry is a duplicate, and “banana ” is kept as separate because “Banana” and “banana” differ in case. Tick “Ignore upper and lower case” and it joins them. The result becomes apple, Banana, cherry, a blank line, a blank line and date, because the blank lines are not touched unless you tick the empty-line options. If you also tick “Remove empty lines”, you get a clean four-line list: apple, Banana, cherry, date.

Everyday uses

  • Cleaning an email or phone list before an upload.
  • Merging keyword lists from several sources.
  • Removing repeated lines from a log file.
  • Tidying a to-do list copied from several notes.
  • Finding which items appear twice in a reading list.

Common mistakes

  • Forgetting case. Email addresses are usually treated as the same regardless of case in the domain part, but this tool compares exactly unless you tick the case option.
  • Hidden spaces. Copying from spreadsheets can add trailing spaces or non-breaking spaces. Trim and check.
  • Expecting near matches. “Colour” and “Color” are different lines. The tool finds exact matches only, after your clean-up options.
  • Losing order. The result keeps your original order. If you also need it sorted, use the sort text lines tool.
  • Not saving the original. Keep your input copy until you are sure the result is right.

Before and after you clean

It helps to look at your data before you clean it. Scroll through the input and notice patterns: do some lines have extra spaces, do some end with a comma, are there blank lines between groups? Those patterns decide which options to tick. After cleaning, compare the counts. If you had 200 lines and 40 were removed, that is a reasonable result for a list built from several sources. If 190 were removed, something is probably wrong, for example a cleaning option that made many different lines look the same. When in doubt, run the tool once with only trimming on, check the result, and then add one option at a time.

Safety, limits and privacy

The tool uses a plain lookup table without prototype keys, so lines such as “constructor” or “__proto__” are handled as ordinary text. Input is limited to about four million characters to keep your browser responsive. All processing happens in your browser and nothing is uploaded, which matters when your list contains names or email addresses.

Remember that removing duplicates from personal data does not make it safe to share. Follow your local data protection rules. For more on tidy data habits, see the Excel course, the post on organising files and folders and the text comparison tool, which shows exactly what differs between two lists.

Key takeaways

  • Duplicates are found by exact match after your clean-up options.
  • Trim spaces and choose case handling to match your data.
  • The first copy is kept unless you choose the last.
  • Your list stays in your browser.

Questions

Does the tool keep the original order?

Yes. It keeps your lines in their original order and only removes later repeats, or earlier ones if you choose to keep the last copy. If you want an alphabetical list afterwards, run the result through the sort tool, which can also remove duplicates in the same step.

Why are two identical-looking lines not removed?

Usually because of a hidden difference such as a trailing space, a different letter case or a non-breaking space copied from a web page. Switch on trimming and ignore-case options and try again. The repeated-lines table shows what was matched, so you can see what is left.

Can I choose which duplicate to keep?

Yes. The default keeps the first copy of each line. Choose "The last one" to keep the final copy instead, which is useful when later lines are updates of earlier ones. In both cases the lines that remain stay in their original order.

Will it remove near-duplicates like Color and Colour?

No. It matches lines exactly, after the clean-up options you pick, such as trimming spaces and ignoring case. Spelling variants, typos and lines that differ by one word are treated as different. For those, use the text comparison tool to look at differences yourself.

Is my list uploaded?

No. All processing happens in your browser and nothing is sent or stored. That is useful for lists of names or email addresses. Even so, follow your local rules for handling personal data, and keep a copy of your original list until you have checked the result.