Merging and Sorting Files with uniq Command

Sean Hannity

less than a minute read

·

09-11-2024

11

0

Share

The uniq command is a powerful utility in Unix and Linux systems that is primarily used to filter out duplicate lines in a text file. It is often combined with other commands such as sort to manage and organize data efficiently. This article will provide an overview of how to merge and sort files using the uniq command.

Basic Usage of uniq

The basic syntax of the uniq command is as follows:

uniq [OPTION]... [INPUT [OUTPUT]]

Key Options

• -c: Prefix lines by the number of occurrences.
• -d: Only print duplicate lines.
• -u: Only print unique lines.
• -i: Ignore case differences.

Sorting Files

Before using uniq, it is essential to sort the files. The sort command organizes the lines of a file in a specified order, making it easier to identify duplicates. Here's how to use it:

sort filename.txt

You can also sort multiple files by listing them one after the other:

sort file1.txt file2.txt > sorted_file.txt

Merging Files and Removing Duplicates

To merge multiple files and remove duplicate lines, follow these steps:

1. Sort the Files: First, sort the files you wish to merge.
2. Pipe to uniq: Use the output from sort and pipe it to uniq.

Example Command

sort file1.txt file2.txt | uniq > merged.txt

In this example, file1.txt and file2.txt are sorted and then merged into merged.txt, where all duplicate lines are removed.

Practical Example

Let’s look at a practical example:

Assume file1.txt contains:

apple
banana
cherry
banana

And file2.txt contains:

banana
cherry
date
elderberry

Step 1: Merge and Sort

To merge and sort these files while removing duplicates:

sort file1.txt file2.txt | uniq > merged_sorted.txt

Step 2: Check the Output

The merged_sorted.txt will contain:

apple
banana
cherry
date
elderberry

Summary

Using the uniq command in combination with sort allows users to efficiently merge files and eliminate duplicate entries. This is particularly useful in data management and processing scenarios where clean, unique entries are required.

For more complex tasks, consider exploring additional options provided by the uniq command to customize its behavior according to your needs.