Guide
sort and uniq
A practical TinyDevTools guide.
What uniq actually does
Unix uniq primarily collapses adjacent identical lines. If identical lines are separated, a plain uniq does not remove them.
alpha beta alpha uniq keeps all three. sort | uniq alpha alpha beta becomes alpha beta
Why sorting changes the problem
sort groups equal lines so uniq can collapse them. The tradeoff is that original order is discarded. That is often useful for set-like data but not for chronological or priority-ordered data.
Choosing the workflow
- Use stable deduplication when first-seen order matters.
- Use sorting when canonical order is the desired output.
- Use a duplicate finder when you need to inspect repetition before changing anything.