Mastering Linux Bash Redirection: Pipes, Tee, and Streams

When you’re managing production servers, you spend a lot of time moving data between commands. I remember once spending about two hours trying to debug a log parsing script, only to realize I was blindly dumping everything into a file instead of filtering it on the fly. Mastering linux bash redirection is the difference between writing messy, temporary files and building elegant, one-line pipelines.

Understanding Streams: stdin, stdout, and stderr

Every process in Linux starts with three standard data channels, or "streams":

1. stdin (0): Standard input. This is where your command reads data (like typing into your keyboard).
2. stdout (1): Standard output. This is where your command prints the successful result.
3. stderr (2): Standard error. This is where the command sends error messages and warnings.

The magic happens when you start pointing these streams somewhere else.

Redirecting Output and Errors

If you want to save the output of a command to a file, you use the > operator. Be careful: > overwrites the file, while >> appends to it.

Bash
# Save output to a log file
ls -l /etc > server_files.txt

# Append errors to a separate log
grep "error" /var/log/syslog 2>> error_log.txt

One of my favorite tricks is redirecting both output and errors to the same file. You’ll often see this in crontab jobs:

Bash
# Redirect stderr (2) to stdout (1), then send both to the file
./deploy.sh > deploy.log 2>&1

The Power of the Pipe Command Linux

The pipe command linux (|) is the backbone of the shell. It connects the stdout of the first command directly to the stdin of the second.

I use this constantly to filter data. Instead of saving a massive list of processes to a file and searching through it, I just pipe it:

Bash
# Find a specific running process
ps aux | grep "nginx"

If you are new to this, it helps to visualize how these tools interact.

Using Tee for Real-Time Logging

Sometimes you want to see the output on your screen and save it to a file simultaneously. That’s where the tee command examples come in handy. It acts like a T-junction in a pipe.

Bash
# See the output and save it to a file
dmesg | tee kernel_boot.log | grep "error"

In this case, dmesg sends its output to tee, which writes a copy to kernel_boot.log while passing the original data down the pipe to grep. It’s incredibly useful for debugging long-running scripts where you don't want to lose the output if the shell crashes.

Practical Tips for Bash Shell Scripting

When you're writing scripts, remember that your shell environment matters. If you're struggling with permissions or ownership during these operations, you might need to brush up on Linux File Permissions: Mastering ACLs and Sticky Bits to ensure your user has the right to write to those log files.

Also, keep your scripts clean. If you're building automated tools, consider structuring them properly, much like how we handle Mastering Laravel Artisan Command Development for CLI Tools to keep CLI logic separate from application logic.

FAQ

What is the difference between | and >? The pipe (|) sends the output of one command to another program for processing. The redirect (>) sends the output of a command to a file.

How do I discard output I don't want? Redirect it to /dev/null. It’s a special "black hole" file: command > /dev/null 2>&1.

Why does my pipe stop working? Check if your command is actually writing to stdout. Some programs (like certain interactive tools) write directly to the terminal device, which a pipe cannot capture.

Next time you're working, try to avoid creating temporary files. If you find yourself writing a file just to read it again in the next line, you're probably ready to refactor that into a pipe. It's cleaner, faster, and keeps your disk usage down. I still occasionally catch myself creating unnecessary temp files during quick debugging sessions—it's a habit that takes a while to break.