How Unix sort and uniq work together

The common sort | uniq pipeline works because uniq compares neighboring lines. Understanding that step helps you choose between a sorted set, an occurrence count, and first-seen deduplication.

What uniq actually does

Unix uniq primarily collapses adjacent identical lines. If identical lines are separated, a plain uniq does not remove them.

alpha
beta
alpha

uniq keeps all three.

sort | uniq

alpha
alpha
beta

becomes

alpha
beta

Why sorting changes the problem

sort groups equal lines so uniq can collapse them. The tradeoff is that original order is discarded. That is often useful for set-like data but not for chronological or priority-ordered data.

Choose an explicit output

For a line-oriented file where sorted order is acceptable, these GNU/Linux commands illustrate different results. They assume GNU coreutils and an existing names.txt; the C locale makes the ASCII example's ordering reproducible:

printf '%s\n' beta alpha beta > names.txt
LC_ALL=C sort -- names.txt | uniq
LC_ALL=C sort -- names.txt | uniq -c
LC_ALL=C sort -u -- names.txt

The first command outputs alpha and beta; the count form reports one alpha and two beta lines; the last outputs the same sorted unique values. GNU uniq counts or removes adjacent identical lines; sorting groups equal lines first. Count-column spacing is for alignment and can vary with the largest count.

sort has options that affect comparison, locale, and numeric behavior. The examples use lowercase ASCII; for mixed-case or non-ASCII data, explicitly choose the ordering your workflow requires and consult your local manual. These options are shown for GNU/Linux and may need adjustment on other Unix-like systems. See the GNU uniq manual for exact option behavior.

When plain uniq is enough

If duplicates are already adjacent and you want to collapse only consecutive repeats, plain uniq preserves the order of the remaining groups. It does not find equal values separated by other lines. This can be useful for run-length style summaries, where transitions and group boundaries matter.

Input:
red
red
blue
red

uniq output:
red
blue
red

Protect source data and choose semantics

Do not redirect output over the same source file you are reading. Write to a separate file, inspect the output, and replace the source only when you are certain. For a stable first-seen result, use a stable-deduplication method instead of sorting; for investigation, count or find duplicates before changing anything.

Choosing the workflow

Inspect duplicates first →

Related: preserve first-seen order in the browser or learn the surrounding Linux text-file commands.

Advertisements