I did a year in network operations at Spark back in New Zealand, and the lesson that stuck was simple: people make mistakes and systems fail, on their own schedule, without asking whether you’re ready. The only variable you control is how prepared you are when it happens.

Make mistakes safe to admit: the blameless postmortem

#

The biggest single improvement isn’t technical. If people get punished for mistakes, they hide them, and a hidden mistake gets fixed hours or days later than it should. A team where someone can say “I broke it, here’s exactly what I did” within minutes will always recover faster. The follow-up matters just as much: don’t stop at patching the symptom. Dig for the root cause, or the same mistake comes back wearing a different hat.

That follow-up is what a blameless postmortem is for. After any incident worth learning from, someone writes it up: what happened, what it cost, how it was fixed. The write-up assumes everyone acted in good faith with what they knew at the time, so instead of asking who to blame, it asks what made the mistake easy: a missing confirmation step, a misleading dashboard, a runbook that didn’t exist. “An engineer ran a migration against production by mistake” becomes “the deploy script let a migration run against production without a check”. Then the write-up is shared, so other teams learn from it too, and it ends with action items, each with an owner and a date, so it doesn’t become a document nobody acts on. The postmortem chapter of Google’s SRE book, Postmortem Culture: Learning from Failure, is the long version, and it comes with a worked example.

A blameless postmortem template

#

Something like this is enough to start with. Keep it short enough that people actually fill it in.

# Postmortem: <short title>
Date: YYYY-MM-DD · Authors: <names> · Status: draft / reviewed

## Summary
Two or three sentences: what broke, for how long, how it was fixed.

## Impact
Who was affected, how many, for how long. Any lost data or revenue.

## Timeline
One time zone throughout.
- 14:02 deploy starts
- 14:09 first alert fires
- 14:15 rollback starts
- 14:21 error rate back to normal

## Contributing factors
What in the system, process or tooling let this happen. No names.

## What went well

## What didn't go well

## Action items
| Action                                      | Owner  | Due        |
|---------------------------------------------|--------|------------|
| Add a confirmation step to migration script | <name> | YYYY-MM-DD |

Reduce the odds of the mistake in the first place

#

Three boring things do most of the work here. Keep skills current — errors love outdated knowledge, so regular training is maintenance, not a perk. Write procedures down — a lot of “human error” is really ambiguity error, someone guessing because the doc didn’t exist. And automate the repetitive stuff: humans are terrible at doing the same task correctly five hundred times, and computers are built for exactly that. On top of those, put a second pair of eyes on anything risky. Peer review, or even a structured self-check, catches an embarrassing share of problems before they ship.

Plan for the big one before it arrives

#

Start with an honest risk assessment: internal weak points like your IT infrastructure, external ones like natural disasters. Put real-time monitoring in place as your early-warning system. Then write the plans: a business continuity plan for disruptions in general, and a disaster recovery plan for the IT side, which says how long each system can be down and how much data you can afford to lose (the recovery time and recovery point objectives, RTO and RPO).

An untested plan is a guess, so run drills. Decide in advance who tells stakeholders what, because improvised crisis communication is how panic spreads. And don’t skip the unglamorous rest: stay in touch with local authorities, keep up maintenance, carry the right insurance.

None of this is exciting, which is probably why it works. The companies that handle failures well aren’t lucky — they planned for them while everything was still fine.