Pipelines and filters
Chain small tools with | to sort, count, slice and transform text in one line.
- Connect commands with pipes and reason about the text between stages
- Use sort, uniq, head, tail, wc, cut and tr as building blocks
- Write the classic frequency count: sort | uniq -c | sort -nr
A pipe, |, connects one command’s stdout to the next command’s stdin. The commands run at the same time, text streaming between them, and no temporary files are needed:
cat station.log | grep ERROR | wc -lRead it left to right: take the log, keep the ERROR lines, count them. Commands that read text, change it and write text are called filters, and a handful of them cover an astonishing amount of everyday work:
| Filter | Does | Useful options |
|---|---|---|
sort | sorts lines | -n numeric, -r reverse, -u unique, -t, -k2 by the 2nd comma-separated field |
uniq | collapses adjacent duplicate lines | -c prefix each with its count |
head / tail | first / last lines | -n 5 |
wc | counts | -l lines, -w words |
cut | picks columns | -d" " -f3 the 3rd space-separated field, -c1-5 characters |
tr | translates characters | tr a-z A-Z upper-case, -d delete, -s squeeze repeats |
Try it
Pipeline lab
Here’s a morning’s worth of station.log: each line has a time, a level, a subsystem and a message. Click commands to build a pipeline and watch the text after every stage. Match each target output.
Notice in challenge 2 what happens if you use uniq before sort.
Challenge 1: How many ERROR lines are in the log?
$ cat station.log
Click a command to add it to the end of the pipeline:
08:00 INFO life-support oxygen nominal 08:05 WARN reactor temperature rising 08:09 ERROR reactor coolant pump stalled 08:12 INFO galley coffee brewed 08:20 ERROR comms antenna misaligned 08:31 INFO reactor temperature stable 08:40 ERROR reactor coolant pump stalled 08:42 WARN comms signal weak 08:55 ERROR galley coffee machine jammed 09:03 INFO galley coffee brewed 09:10 ERROR reactor pressure spike 09:15 WARN life-support filter clogged
5
Challenge 3 is worth memorizing - it answers “what are the most common X?” for any X:
... | sort | uniq -c | sort -nr | headThe first sort puts identical lines next to each other, uniq -c collapses each run into one line with its count, sort -nr orders by that count, biggest first, and head keeps the top ten.
1cat > crew.csv <<'EOF'
2ada,engineer,deck2
3grace,captain,deck1
4linus,engineer,deck2
5margaret,pilot,deck1
6ken,engineer,deck3
7EOF
8echo "Roles by headcount:"
9cut -d, -f2 crew.csv | sort | uniq -c | sort -nr
10echo "Decks:"
11cut -d, -f3 crew.csv | sort -u | tr '\n' ' '
12echo
13echo "First two names, upper-cased:"
14head -n 2 crew.csv | cut -d, -f1 | tr a-z A-ZRoles by headcount:
3 engineer
1 pilot
1 captain
Decks:
deck1 deck2 deck3
First two names, upper-cased:
ADA
GRACENotice tr '\n' ' ': it turns newlines into spaces, joining lines into one. Also notice the ties: pilot and captain both have 1, and GNU sort breaks the tie by comparing whole lines, here in reverse because of -r. (Sorting also depends on your locale: in your terminal, LC_ALL=C sort gives the same plain byte order as this sandbox.)
Key takeaways
|streams one command’s stdout into the next command’s stdin.Small filters - sort, uniq, head, tail, wc, cut, tr - combine into powerful one-liners.
sort | uniq -c | sort -nrcounts and ranks anything.uniqneeds sorted input;sort -uis a shortcut when you don’t need counts.
Lesson quiz
7 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Bash scripts
Write a script in the editor and run it for real against sample input. Each run gets a fresh Linux sandbox with Bash 5.2 and the GNU tools on Wandbox, a free public service - so experiment freely, even with rm. Your script and test input are sent there.
Airlock regulars
Your script’s stdin is the airlock access log: one crew name per line, in the order they cycled through. Print the three names that used the airlock most, most frequent first, in uniq -c format.
- A busy morning
- A quiet morning
Your script runs with Bash 5.2 and GNU tools on Wandbox, a free public service, in a fresh sandbox each time. Your script and test input are sent to that service.
Roster roles
stdin is the crew roster as CSV: name,role,deck. Print each role once, in capitals, sorted alphabetically.
- Five crew
- New arrivals
Your script runs with Bash 5.2 and GNU tools on Wandbox, a free public service, in a fresh sandbox each time. Your script and test input are sent to that service.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…