Pipes and Redirection🔗

Part of Essentials

This article assumes you're comfortable with basic commands and command chaining (&&, ||). Pipes are related but different — they connect command output to command input, not just sequence execution.

The most powerful Linux one-liners you'll ever see don't use a single complex command. They use simple commands connected together. cat access.log | grep "500" | awk '{print $1}' | sort | uniq -c | sort -nr | head -10 — that's ten minutes of investigation distilled to one line, and each part is a simple tool you already know.

That's the Unix philosophy in action: programs that do one thing well, connected by pipes. This article shows you how to connect them.

Where You've Seen This🔗

If you've used PowerShell, you know pipes — Get-Process | Where-Object CPU -gt 10 | Sort-Object CPU -Descending. The Linux version works the same way conceptually, but passes plain text instead of objects. That difference matters: Linux pipelines are more composable (any tool can talk to any other tool), but you work with structured text rather than typed objects.

If you've built ETL pipelines, data workflows, or streaming architectures — Kafka, Spark, Logstash — the stdin/stdout model is the same idea at shell scale. Data flows from source through transformations to sink. Each command is a transform.

Redirection (>, >>, 2>) is the command-line equivalent of writing to a file in any language. You've done this before; the Linux operators are just a more concise syntax.

Standard Streams: The Foundation🔗

Every process in Linux has three data channels open by default:

graph LR
    STDIN["📥 stdin (0)\nStandard Input\n← Keyboard or pipe"] --> PROCESS["⚙️ Command\nor Process"]
    PROCESS --> STDOUT["📤 stdout (1)\nStandard Output\n→ Screen or redirect"]
    PROCESS --> STDERR["⚠️ stderr (2)\nStandard Error\n→ Screen or redirect"]

    style STDIN fill:#2d3748,stroke:#63b3ed,stroke-width:2px,color:#fff
    style PROCESS fill:#d69e2e,stroke:#cbd5e0,stroke-width:2px,color:#000
    style STDOUT fill:#2d3748,stroke:#68d391,stroke-width:2px,color:#fff
    style STDERR fill:#2d3748,stroke:#fc8181,stroke-width:2px,color:#fff

Stream	Number	Default	What Goes Here
stdin	0	Keyboard	Input to commands
stdout	1	Terminal	Normal output
stderr	2	Terminal	Error messages

By default, stdout and stderr both print to your terminal mixed together. Redirection and pipes give you control over each stream separately.

Redirection: To and From Files🔗

Redirection changes where a stream goes — from the terminal to a file, or from a file to a command's input.

Output Redirection (>)

Why it matters: Capture command output to a file for logging, later processing, or sharing with others.
Redirect stdout to a File
```
ls -lh /var/log/ > filelist.txt   # (1)!
df -h > disk-report.txt
```
1. Create or overwrite the file.
> Overwrites Without Warning

> will silently overwrite an existing file. There's no confirmation. If report.txt existed, it's gone.

To protect against this in interactive sessions:
Prevent Overwrite with noclobber
```
set -o noclobber    # (1)!
ls > existing.txt   # (2)!
ls >| existing.txt  # (3)!
```
1. Add this to ~/.bashrc.
2. Now fails with "cannot overwrite existing file".
3. Force the overwrite when noclobber is set.
Append Redirection (>>)

Why it matters: Add to an existing file without overwriting it. The essential operator for log files and accumulating data.
Append to a File
```
echo "Deployment started at $(date)" >> deploy.log
df -h >> disk-history.txt   # (1)!

echo "=== $(date) ===" >> report.txt   # (2)!
uptime >> report.txt
free -h >> report.txt
df -h >> report.txt
```
1. Build a history over time.
2. Run multiple commands and collect all their output into one report.
Input Redirection (<)

Why it matters: Feed a file into a command's stdin instead of typing. Less common in interactive use, but appears in scripts.
Redirect File to stdin
```
sort < names.txt                     # (1)!
wc -l < access.log                   # (2)!
mysql -u root -p mydb < schema.sql   # (3)!
```
1. Sort the contents of names.txt.
2. Count lines in access.log.
3. Feed SQL to mysql.
stderr Redirection (2>)

Why it matters: Error messages and normal output are separate streams. When you redirect with >, errors still print to the terminal. Use 2> to capture them separately.
Redirect stderr
```
find / -name "hosts" 2>/dev/null      # (1)!
find / -name "hosts" 2>errors.txt     # (2)!
command > output.txt 2> errors.txt    # (3)!
command > all.txt 2>&1                # (4)!
```
1. Discard errors, keep results.
2. Save errors to a file.
3. Separate stdout and stderr into different files.
4. 2>&1 redirects stderr (2) to wherever stdout (1) is currently going. Order matters: > all.txt 2>&1 works; 2>&1 > all.txt does not (the latter redirects stderr to the terminal, then redirects stdout to the file).

Combining Redirections🔗

Multiple Redirections in Practice

find /var/log -name "*.log" | tee found-logs.txt    # (1)!
./long-running-script.sh > results.txt 2>/dev/null  # (2)!
./deployment.sh > deploy.log 2>&1                    # (3)!
./script.sh > output.log 2> errors.log               # (4)!

Capture output to a file and show it on screen at once — tee reads stdin and writes to both stdout and a file.
Run a long job, save the output, discard errors.
Save both stdout and stderr to the same file.
Save stderr to one file, stdout to another.

Here Documents and Here Strings🔗

For feeding multi-line input to commands:

Here Documents

cat << EOF > /etc/myapp/config.conf   # (1)!
[database]
host = db-prod-01
port = 5432
name = myapp_db
EOF

sudo tee /etc/nginx/sites-available/myapp << EOF   # (2)!
server {
    listen 80;
    server_name myapp.example.com;
    root /var/www/myapp;
}
EOF

Feed multiple lines to a command — everything up to the EOF marker becomes stdin.
Common pattern for generating config files in scripts.

Here String (<<<)

grep "pattern" <<< "string to search in"   # (1)!
base64 <<< "encode this text"               # (2)!

Feed a single string to stdin.
Practical use — feed a variable to a command that expects stdin.

Pipes: Command to Command🔗

A pipe (|) takes the stdout of the left command and feeds it as stdin to the right command. No intermediate file, no waiting — it's streaming.

Your First Pipeline

ls -lh /var/log/ | less   # (1)!

ls produces the directory listing, the pipe (|) sends it to less, and less lets you scroll through it.

Building Pipelines🔗

Pipelines chain as many commands as you need:

Growing a Pipeline

cat /var/log/nginx/access.log # (1)!
cat /var/log/nginx/access.log | grep " 500 " # (2)!
cat /var/log/nginx/access.log | grep " 500 " | awk '{print $1}' # (3)!
cat /var/log/nginx/access.log | grep " 500 " | awk '{print $1}' | sort # (4)!
cat /var/log/nginx/access.log | grep " 500 " | awk '{print $1}' | sort | uniq -c # (5)!
cat /var/log/nginx/access.log | grep " 500 " | awk '{print $1}' | sort | uniq -c | sort -nr # (6)!
cat /var/log/nginx/access.log | grep " 500 " | awk '{print $1}' | sort | uniq -c | sort -nr | head -10 # (7)!

Start simple — dump the whole access log.
Filter to 500 errors only.
Extract just the IP addresses.
Sort them.
Count occurrences of each IP.
Sort by count, most first.
Show only the top 10.

This is how investigation pipelines get built — one stage at a time, checking the output at each step.

The Essential Pipeline Tools🔗

These commands exist primarily to work within pipelines:

grep — Filter Lines

Filter a stream to only lines matching a pattern. Covered in depth in the grep article.
grep in Pipelines
```
journalctl | grep "Failed"
ps aux | grep nginx
cat /etc/passwd | grep -v "nologin"    # (1)!
```
1. Exclude service accounts.

sort — Sort Lines

Sort lines alphabetically or numerically.

sort in Pipelines

cat names.txt | sort              # (1)!
du -sh /var/* | sort -hr          # (2)!
cat numbers.txt | sort -n         # (3)!
cat file.txt | sort -u            # (4)!

Alphabetical.
Human-readable sizes, largest first.
Numeric sort.
Sort and remove duplicates.

uniq — Remove Duplicates / Count

Remove consecutive duplicate lines, or count their occurrences. Works best after sort.
uniq in Pipelines
```
cat file.txt | sort | uniq               # (1)!
cat file.txt | sort | uniq -c            # (2)!
cat file.txt | sort | uniq -d            # (3)!
```
1. Unique lines only.
2. Count occurrences.
3. Only lines that appear more than once.

wc — Count

Count lines, words, or characters.

wc in Pipelines

ls /etc | wc -l                      # (1)!
cat access.log | grep "404" | wc -l  # (2)!
cat file.txt | wc -c                 # (3)!

How many files in /etc?
How many 404 errors?
How many bytes?

awk — Extract Fields

Extract specific columns from structured text. The most powerful field processor in the pipeline toolkit.
awk in Pipelines
```
ps aux | awk '{print $1, $2}'                # (1)!
df -h | awk '{print $1, $5}'                 # (2)!
cat /etc/passwd | awk -F: '{print $1, $3}'   # (3)!
```
1. Print the user and PID columns.
2. Filesystem and usage %.
3. Username and UID.
cut — Extract Columns

Extract specific columns or character positions from fixed-format output.
cut in Pipelines
```
cat /etc/passwd | cut -d: -f1     # (1)!
cat /etc/passwd | cut -d: -f1,3   # (2)!
ls -l | cut -c1-10                # (3)!
```
1. Extract the first field (username).
2. Username and UID.
3. First 10 characters of each line.
tee — Split Output

Write to a file AND pass through to the next pipe. Essential when you want to save intermediate results.
tee in Pipelines
```
find /var/log -name "*.log" | tee found-logs.txt | wc -l   # (1)!
command | tee output.txt | grep "ERROR"                    # (2)!
```
1. Saves the file list AND prints the count.
2. Saves everything to output.txt, passes only errors downstream.
head / tail — Limit Output

Take only the first or last N lines. Ubiquitous at the end of pipelines.
head and tail in Pipelines
```
ps aux | sort -k3 -nr | head -5      # (1)!
ls -lt /var/log/ | head -10          # (2)!
cat access.log | tail -100           # (3)!
```
1. Top 5 CPU consumers.
2. 10 most recently modified logs.
3. Last 100 lines.

stderr in Pipelines🔗

A critical detail: pipes only carry stdout. stderr bypasses the pipe and goes directly to the terminal.

Stderr Bypasses Pipes

find / -name "*.conf" | wc -l   # (1)!

Errors print to the screen; only successful results go to wc.

To include stderr in a pipeline:

Including stderr in Pipelines

find / -name "*.conf" 2>&1 | wc -l          # (1)!
find / -name "*.conf" 2>/dev/null | wc -l   # (2)!

Count everything, including error lines.
Suppress errors, count only results.

Real-World Pipeline Patterns🔗

Log AnalysisProcess InvestigationDisk Space InvestigationConfig File Work

Find the most common errors in the past hour:

Log Analysis Pipeline

grep " 404 " /var/log/nginx/access.log \
  | awk '{print $1}' \
  | sort | uniq -c \
  | sort -nr \
  | head -10                              # (1)!

journalctl --since "24 hours ago" --until now \
  | grep -i "error" \
  | awk '{print $1, $2, substr($3,1,2)}' \
  | sort | uniq -c                        # (2)!

What IP addresses are causing the most 404s?
How many errors per hour in the last day?

Find what's consuming resources:

Process Investigation Pipeline

ps aux --sort=-%mem | head -6   # (1)!

ps aux | grep "^www-data"       # (2)!

ss -tlnp | grep ":8080"         # (3)!

ss -s | grep "TCP:"             # (4)!

Top 5 memory consumers (human-readable).
All processes owned by www-data.
Find processes listening on a port.
How many connections per state?

Find what's eating disk:

Disk Analysis Pipeline

du -sh /var/* 2>/dev/null | sort -hr | head -10   # (1)!

find / -type f -size +100M 2>/dev/null \
  | xargs ls -lh 2>/dev/null \
  | sort -k5 -hr \
  | head -10                                      # (2)!

find /var/log -name "*.log" -newer /var/log -type f \
  | xargs ls -lh 2>/dev/null \
  | sort -k5 -hr                                   # (3)!

Largest directories in /var.
Largest files anywhere on the system.
Which log files grew today?

Process configuration files:

Config Processing Pipeline

grep -v "^#" /etc/ssh/sshd_config | grep -v "^$"   # (1)!

grep -v "^#" /etc/ssh/sshd_config \
  | grep -v "^$" \
  | awk '{print $1}' \
  | sort -u                                         # (2)!

grep -v "^#" /etc/ssh/sshd_config | grep "PermitRootLogin"   # (3)!

Find all uncommented settings in a config file.
Find all unique setting names.
Check if a setting is enabled.

Quick Reference🔗

Redirection Operators🔗

Operator	What It Does	Example
`>`	Redirect stdout to file (overwrite)	`ls > files.txt`
`>>`	Redirect stdout to file (append)	`echo "line" >> log.txt`
`<`	Redirect file to stdin	`sort < names.txt`
`2>`	Redirect stderr to file	`find / 2>/dev/null`
`2>&1`	Redirect stderr to stdout	`command > all.txt 2>&1`
`&>`	Redirect both stdout and stderr	`command &> all.txt`
`\\|`	Pipe stdout to next command	`ls \\| grep ".conf"`
`\\| tee`	Write to file and pass through	`cmd \\| tee file.txt \\| wc -l`

Pipeline Toolkit🔗

Command	What It Does in Pipelines
`grep pattern`	Keep only lines matching pattern
`grep -v pattern`	Remove lines matching pattern
`sort`	Sort lines alphabetically
`sort -n`	Sort numerically
`sort -hr`	Sort human-readable sizes, largest first
`uniq`	Remove consecutive duplicates
`uniq -c`	Count occurrences
`wc -l`	Count lines
`head -N`	Keep first N lines
`tail -N`	Keep last N lines
`awk '{print $N}'`	Extract Nth field
`cut -d: -f1`	Extract field by delimiter
`tee file.txt`	Write to file and pass through

Practice Exercises🔗

Exercise 1: Build a Log Analysis Pipeline

Using /var/log/auth.log (or /var/log/secure on RHEL), build a pipeline that:

Shows only lines containing "Failed"
Extracts the source IP address (the IP after "from" in the line)
Counts how many failed attempts per IP
Sorts by count, highest first
Shows the top 5 offenders

Solution

Log Analysis Pipeline

grep "Failed" /var/log/auth.log \
  | awk '/from/{print $(NF-3)}' \
  | sort \
  | uniq -c \
  | sort -nr \
  | head -5

The awk pattern prints the field 3 from the end (NF-3) — on typical auth.log "Failed password" lines, this is the IP address.

Exercise 2: Redirect and Tee

Run a disk space analysis (du -sh /var/* 2>/dev/null) that:

Saves the complete output to /tmp/disk-report.txt
Simultaneously shows only entries larger than 1GB on the terminal

Solution

tee with Filtering

du -sh /var/* 2>/dev/null | tee /tmp/disk-report.txt | grep "^[0-9]*G"

tee saves everything to the file; the pipe after tee sends the same data to grep, which filters for lines starting with a size in gigabytes.

Exercise 3: Separate stdout and stderr

Run find / -name "sshd_config" and:

Save only the successful results (stdout) to /tmp/found.txt
Discard all "Permission denied" errors (stderr)
Print the count of found files to the terminal

Solution

Separate Streams

find / -name "sshd_config" 2>/dev/null > /tmp/found.txt
wc -l < /tmp/found.txt

find / -name "sshd_config" 2>/dev/null | tee /tmp/found.txt | wc -l   # (1)!

Or do it in one pipeline — tee saves the results while wc -l counts them.

Quick Recap🔗

Three streams: stdin (0), stdout (1), stderr (2) — redirect each independently
> overwrites; >> appends — the most common mistake is using > when you meant >>
2>/dev/null — silence errors; ubiquitous in find and other commands that hit permission-denied directories
2>&1 — merge stderr into stdout; essential when capturing all output to a file
| pipes only stdout — to include stderr, add 2>&1 before the pipe
Build pipelines incrementally — add one stage at a time, verify output, then extend
Core toolkit: grep, sort, uniq -c, wc -l, head, tail, awk, tee

What's Next?🔗

Pipes flow data between commands, and the most common thing you'll do with that data is search it. grep is the workhorse of Linux text processing — and it deserves its own deep dive.

Head to grep to learn regular expressions, recursive searching, context flags, and the patterns that turn grep from a simple filter into a powerful investigation tool.

Pipes and Redirection🔗

Where You've Seen This🔗

Standard Streams: The Foundation🔗

Redirection: To and From Files🔗

Combining Redirections🔗

Here Documents and Here Strings🔗

Pipes: Command to Command🔗

Building Pipelines🔗

The Essential Pipeline Tools🔗

stderr in Pipelines🔗

Real-World Pipeline Patterns🔗

Quick Reference🔗

Redirection Operators🔗

Pipeline Toolkit🔗

Practice Exercises🔗

Quick Recap🔗

Further Reading🔗

Command References🔗

Deep Dives🔗

Official Documentation🔗

What's Next?🔗