0x80Lesson 9 of 9

Design for recovery and learn from failures

Combine health signals, graceful degradation, and recovery objectives.

14 min 5-question quiz

By the end of this lesson you can

Distinguish redundancy from resilience and use SLI, SLO, RTO, and RPO appropriately.

Redundancy provides spare capacity, but resilience also needs failure detection, safe failover, tested recovery, and clear operator signals. An SLI is a measured indicator such as successful requests; an SLO is a target for that indicator over a window. RTO is the target time to restore service after disruption, while RPO is the acceptable amount of data loss measured in time. Graceful degradation preserves essential behavior when optional dependencies fail.

A real SLI needs a clear event definition, exclusions, and a measurement window. Pair metrics with logs and traces so teams can find where a request slowed or failed.

Practice: use the right word

Try it

Up to 15 XP

SLI, SLO, RTO or RPO?

These four terms get confused all the time. Sort each statement.

0 of 6 sortedScore 0/0

“The share of checkout requests that succeed.”
“99.9% of requests succeed over 30 days.”
“After a disaster, service must be back within 2 hours.”
“We can afford to lose at most 15 minutes of orders.”
“p95 latency of search requests.”
“Back up every 15 minutes, because of what we can afford to lose.”

Try it

Up to 15 XP

Degrade or fail?

One dependency is down. Decide what a well-designed online store should do for each one.

0 of 5 sortedScore 0/0

“The recommendations service is down.”
“The review service is down.”
“The payment provider is down.”
“The analytics service is down.”
“The inventory service can’t confirm the last item.”

Try it

Up to 15 XP

Review this incident report

Good reviews lead to better systems. Click what would stop this one from helping.

Outage review (round 1 of 1)Flags found 0/0

Click every suspicious section. There are 3.

The site was down for 47 minutes on Tuesday. Sam caused this by running the wrong command. The failover didn’t work because nobody had tested it since it was set up. Action items: Be more careful in future.

Key takeaways

Measure user-visible reliability with clearly defined SLIs and SLOs.
RTO is recovery time; RPO is acceptable data loss.
Resilience includes tested recovery and graceful behavior, not just extra servers.

Lesson quiz

5 questions · pass with 4 correct · up to 50 XP

Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.

Questions about this lesson

Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.

Loading posts…