Capstone: the station log report

Incoming message from Commander Hopper:

Engineer - every morning I scroll through station.log by hand to work out whether the station is on fire. I would like a report on my screen by 09:30 instead. Make it so.

Each log line has four parts - a time, a level (INFO, WARN or ERROR), a subsystem, and a free-text message:

08:09 ERROR reactor coolant pump stalled

The finished tool will print:

1=== KESTREL LOG REPORT ===
2Entries: 12 (08:00-09:15)
3ERROR: 5  WARN: 3  INFO: 4
4Busiest: reactor (5 entries)
5Latest error: 09:10 reactor: pressure spike
6Status: RED

You’ll build it in three exercises. First a quick pipeline, then a loop with associative arrays, then the full report.

Before writing a big script, find out how far one-liners get you. They’re quick to try and easy to check. Counting lines per level is the frequency-count pattern from the pipelines lesson, with awk reformatting the result and adding a total:

A pipeline can continue on the next line after a | - handy for long ones. sort -k1,1nr -k2,2 sorts by count (numerically, descending), then by name, so ties come out in a predictable order.

The full report needs several numbers at once: counts per level and per subsystem, the first and last times, and the latest error. Running a separate pipeline for each would read the log many times. Instead, read it once with while read, splitting each line into fields, and keep tallies in associative arrays:

read -r time level subsystem message puts the first three words in their own variables and everything else in message, so multi-word messages survive intact.

To pick the subsystem with the most entries, you could pipe the tallies through sort | head -n 1. But under set -o pipefail, head closing the pipe early can occasionally make sort fail with SIGPIPE - and with set -e, that ends your script. A small loop avoids the problem, and breaks ties alphabetically:

You can now automate almost anything you can type. Some ideas for what to learn next:

• cron and systemd timers - run your report every morning at 09:25.
• ssh and rsync - run commands and copy files on other machines.
• tmux - keep sessions running and split your terminal into panes.
• jq - a sed-and-awk for JSON, the language of most APIs.
• make - describe how to build things once, and rebuild only what changed.

And whenever a script outgrows a page or two, or needs real data structures, consider moving to Python. Knowing when to switch is part of shell wisdom too.