Writing Post-Mortems: A Guide to SRE Learning and Process

Previously in this course, we covered designing for disaster recovery, which taught you how to prepare for the worst-case scenario. This lesson builds on that foundation by teaching you how to analyze and learn from an incident once it actually happens.

In Site Reliability Engineering (SRE), a post-mortem is not a report card; it is a learning tool. If you treat an outage as a "human error," you've failed the process. If you treat it as a systemic flaw, you've started the journey toward reliability.

The Principles of a Blameless Post-Mortem

A blameless post-mortem assumes that everyone involved acted with the best intentions and the information they had at the time. When things break, it is almost never because a developer "made a mistake." It is because the system allowed that mistake to happen and propagate.

To maintain a blameless culture, focus your language on the system:

• Instead of: "John deleted the production database."
• Use: "The production database was deleted due to a lack of multi-factor authentication on the CLI tool and insufficient IAM scoping."

By shifting the focus, you move from "Who do we blame?" to "How do we fix the safety gaps?"

Identifying Root Causes with the 5 Whys

When you sit down to write your report, you need to dig deeper than the surface-level trigger. We use the 5 Whys technique to peel back layers of causality.

Worked Example: A Service Outage

Incident: The payment gateway service returned 500 errors for 20 minutes.

1. Why did it fail? The service ran out of memory (OOM).
2. Why did it run out of memory? The request queue exploded because the database latency spiked.
3. Why did the database latency spike? A long-running analytical query locked the primary table.
4. Why was the analytical query running on the primary? It was routed to the primary because the read-replica failover mechanism failed.
5. Why did the failover fail? The health check logic relied on a hardcoded IP address that was changed during the last infrastructure update.

Root Cause: The health check configuration was tightly coupled to static infrastructure, preventing automatic recovery.

Defining Actionable Follow-up Tasks

An analysis without action is just a history lesson. Every post-mortem must conclude with a list of tickets. These should be categorized into:

1. Prevention: Tasks that stop this exact failure from recurring (e.g., "Add automated failover testing").
2. Detection: Tasks that help you notice this faster (e.g., "Add alerts for high read-replica lag").
3. Mitigation: Tasks that reduce the impact if it happens again (e.g., "Implement a circuit breaker for analytical queries").

The Post-Mortem Template

Hands-on Exercise

Imagine you experienced an incident where a faulty deployment caused an API to return invalid JSON, crashing mobile clients. Using the 5 Whys method, write a brief root cause analysis for this incident. Once you have the root cause, list three specific follow-up tasks to prevent this from happening again.

Hint: Consider why the bad code made it to production and why the client didn't handle the malformed response gracefully.

Common Pitfalls

• The "Human Error" Trap: If your action item is "Retrain the engineer," you have failed. The system should be robust enough that a single person cannot bring it down.
• Ignoring the Timeline: Without precise timing, you cannot see the gaps in your monitoring and alerting processes.
• The "Shelfware" Report: If your post-mortem sits in a folder and is never discussed, you've wasted your time. Schedule a review meeting to ensure the action items are prioritized in the next sprint.

FAQ

Q: How do I handle a post-mortem when a third-party vendor is at fault? A: Even if the root cause is external, your focus should be on how your system handled the dependency. Did you have a fallback? Should you implement circuit breakers to isolate that vendor?

Q: Is a post-mortem necessary for small, internal bugs? A: Not for every bug, but definitely for any incident that impacts users or requires an emergency rollback. This is a core part of the SRE process for maintaining long-term system stability.

Recap

Writing a post-mortem is about systemic learning. By keeping it blameless, applying the 5 Whys, and generating concrete action items, you transform every failure into an opportunity to harden your architecture.

Up next: Finalizing the Design Document.