Remove Duplicate Lines

Clean a pasted line list

1. Paste or type the list into Original lines. Empty input is valid. Use example fills pear, apple, a repeated pear and a blank row, then waits for you to select Remove duplicates.
2. Select Remove duplicates. The tool checks raw and canonical text against 100000 UTF-16 units and checks all input records against 10000 lines before removing any repeats. The canonical limit includes the final LF needed to terminate the last line.
3. Read lines kept, input lines and duplicate lines removed, then inspect the readonly result. Copy result uses the current logical text; clipboard failure selects it for manual copying. Download TXT produces a UTF-8 file named unique-lines.txt. Clear removes the draft and result.

Understand exact matching and output format

Matching compares complete line contents after CRLF and standalone CR endings become LF. It keeps the first occurrence and preserves order. It does not trim spaces, fold case, normalize Unicode, sort or interpret CSV quoting and columns.

Blank content participates like any other line and survives once. A blank line, a line containing one space and a line containing a tab are three distinct contents. The text a and a followed by LF have the same content for this operation.

The result emits one LF after every retained line. This preserves a retained blank row at the end and makes a second pass return the same text with zero removals. Even when no duplicates are removed, the output can normalize endings or add the final LF; it is not a byte-identity check.

Removed lines count surplus occurrences after the first, not duplicate groups. Three copies of a become one copy, with two removals. The input total always equals lines kept plus lines removed, and similar-looking Unicode sequences can remain distinct.

Examples of order, blanks and significant text

Pear, apple, pear and a blank row produce pear, apple and the blank row: four input lines, three kept and one removed. Order follows the first occurrences, not alphabetical order.

The text a, a blank row and another a without a final newline produces a followed by two LFs. One LF ends the a record and the other preserves the retained blank record. Preserving the source's missing final LF would lose that row.

A, a and A keep both A and a, with one removal. A line containing a, a line with a leading space and a line with a trailing space also remain distinct. Empty text has zero records; one LF has one blank record.

Admission limits and exact text

Both raw text and normalized, fully LF-terminated input must fit 100000 UTF-16 units; at most 10000 input records are allowed before deduplication. A 100000-unit draft without its final terminator would need 100001 canonical units and is rejected, not shortened. These units are not bytes or visible characters.

Malformed UTF-16 is rejected rather than silently repaired. Valid supplementary characters and actual replacement characters remain content. Browser text fields already normalize line endings, so the original clipboard or source-file bytes cannot be recovered.

This version accepts pasted plain text, not uploaded files. It has no trim, case-insensitive, sort, selected-column or blank-line-removal options. Downloads use UTF-8 and LF without an added BOM, while clipboard newline conventions can vary by operating system or destination application.

Local list processing

The tool processes the draft locally and does not upload it, put it in URLs or analytics, or save it in browser storage. Ordinary site and asset requests still occur. Input and result are rendered as text, not executed as HTML or code; there is no saved list or share link.