Introduction
In the world of system management and text processing, the commanduniqis an essential tool to identify and remove duplicate lines in ordered files. Although its operation seems simple, its power is revealed when combined with other utilities such assort, greporawk. In this article we will explore in depth how it worksuniq, your most useful options and practical examples you can apply immediately in your daily workflow.
What is uniq and when to use it?
The nameuniqcomes from «Unique» (unique). Its main mission is to read a text entry line by line and, as long as the lines are adjacent and identical, to show only one copy. This means that, for thatuniqact correctly, the input file must be previously ordered; otherwise, separate duplications will not be detected. It is very useful in logs, configuration files, user lists or any scenario where you need to debug repeated data.
Basic syntax
The simplest way to invokeuniqis:
uniq [options] file
If a file is not specified, the command reads from the standard input, which allows it to be easily chained with pipes (|). For example:
sort archivo.txt | uniq
Order the content and then remove the adjacent repetitions.
Most commonly used options
-c: preplaces to each line the number of times it appears.-d: shows only the lines that are duplicated (one copy of each set).-u: shows only lines that do not repeat (unique).-i: Ignore capital and lower capital when comparing.-f N: omits the first N fields when comparing (useful with delimited files).-s N: omits the first N characters of each line.-w N: compare only the first N characters of each line.
Practical examples
1. Remove simple duplicates
Suppose a filenombres.txtwith the following list (already ordered):
AnaAnaLuisLuisLuisMaria
We run:
uniq nomines.txt
Outcome:
AnaLuisMaria
2. Counting occurrences
To know how many times each name appears:
uniq -c nomies.txt
Output:
2 Ana 3 Louis 1 Mary
3. Show only the duplicates
If we are interested only in the names that are repeated:
uniq -d nomines.txt
Outcome:
AnaLuis
4. Ignore capital / lower capital
With a file with case variations:
anaANALuisluisMaria
First we order without distinguishing case and then applyuniq -i:
sort -f nomines.txt | uniq-i
We will get a single entry for each name regardless of capital letters.
Combining uniq with other commands
The true potential ofuniqis manifested in pipes. Some common patterns:
cat archivo | sort | uniq -c | sort -nr: counts frequencies and orders from greater to lesser.grep "error" log.txt | sort | uniq -c: how many times each error message appears.cut -d: -f1 /etc/passwd | sort | uniq -d: detects duplicate users in the password file (useful for security audits).
Tips and tricks
- Always order before use
uniqUnless you're sure the file is already ordered. - Use
uniq -zif you work with inputs separated by null (useful withfind -print0). - Combine
awkto make auniqcustom when you need to compare only certain fields. - In scripts, check the output code:
uniqreturn 0 if successful and 1 if there were writing errors.
Conclusion
The commanduniqis a small but powerful piece of the arsenal of any Linux administrator or command line lover. Its simplicity hides great versatility, especially when combined with tidy and filtered tools. Dominating your options will allow you to debug logs, clean lists and perform data analysis quickly and efficiently. The next time you meet in front of a file with repeated lines, remember thatuniqis ready to leave it clean and tidy.
