Mission: the 3 a.m. outage
You’re on call and the server is down. Diagnose it and bring it back.
- Work through an outage methodically, from symptoms to causes
- Fix full disks, runaway processes and bad permissions
- Explain a fix, and how to stop it happening again
03:07. Your pager goes off. The station’s comms server has stopped responding, and with it, the link to Earth. You’re the on-call engineer, and you’re the only one awake.
Good engineers don’t guess. They look at the evidence, change one thing at a time, and confirm that it worked. There’s more than one thing wrong tonight.
Try it
Up to 15 XPBring comms back online
Use df -h, du -h, ps, ls -l, cat and tail to find each problem. When you’ve fixed everything, run ./start-comms.sh.
Mission 1 of 6
Read the runbook
Pager: “COMMS DOWN.” Start with what the team already knows. How many checks are in the runbook?
Simulated shell. Try ls, or type help to see the commands.
- comms.conf
- data
- contacts.db
- logs
- comms.log
- indexer-dump.bin
- runbook.txt
- start-comms.sh
- Read the runbook
- Find the first error
- Free the disk
- Stop the runaway
- Fix the config
- Restore comms
Try it
Up to 15 XPWrite the follow-up
Fixing it is half the job. Decide which of these actions would stop this happening again.
“Alert when the disk passes 80%, before it’s a crisis.”
“Limit where the indexer can write, and rotate its output.”
“Tell the on-call engineer to be more careful.”
“Keep the start script’s permissions in version control.”
“Delete the runbook, since it’s now out of date.”
Key takeaways
Read the runbook and the logs before changing anything.
Find the first error, and change one thing at a time.
A fix isn’t finished until you’ve stopped it happening again.
Lesson quiz
4 questions · pass with 3 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Bash scripts
Write a script in the editor and run it for real against sample input. Each run gets a fresh Linux sandbox with Bash 5.2 and the GNU tools on Wandbox, a free public service - so experiment freely, even with rm. Your script and test input are sent there.
Triage the logs
stdin is a server log of TIME LEVEL message lines. Print first error: TIME, using the first ERROR line, then errors: N. If there are no errors, print just all clear.
- An outage
- A quiet night
Your script runs with Bash 5.2 and GNU tools on Wandbox, a free public service, in a fresh sandbox each time. Your script and test input are sent to that service.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…