I did a year in network operations at Spark back in New Zealand, and the lesson that stuck was simple: people make mistakes and systems fail, on their own schedule, without asking whether you’re ready. The only variable you control is how prepared you are when it happens.
Make mistakes safe to admit#
The biggest single improvement isn’t technical. If people get punished for mistakes, they hide them, and a hidden mistake gets fixed hours or days later than it should. A team where someone can say “I broke it, here’s exactly what I did” within minutes will always recover faster. The follow-up matters just as much: don’t stop at patching the symptom. Dig for the root cause, or the same mistake comes back wearing a different hat.
Reduce the odds of the mistake in the first place#
Three boring things do most of the work here. Keep skills current — errors love outdated knowledge, so regular training is maintenance, not a perk. Write procedures down — a lot of “human error” is really ambiguity error, someone guessing because the doc didn’t exist. And automate the repetitive stuff: humans are terrible at doing the same task correctly five hundred times, and computers are built for exactly that. On top of those, put a second pair of eyes on anything risky. Peer review, or even a structured self-check, catches an embarrassing share of problems before they ship.
Plan for the big one before it arrives#
Start with an honest risk assessment — internal weak points like your IT infrastructure, external ones like natural disasters — and put real-time monitoring in place as your early-warning system. Then write the plans: a business continuity plan for disruptions in general, a disaster recovery plan for the IT side specifically. An untested plan is a guess, so run drills. Decide in advance who tells stakeholders what, because improvised crisis communication is how panic spreads. And don’t skip the unglamorous rest: stay in touch with local authorities, keep up maintenance, carry the right insurance.
None of this is exciting, which is probably why it works. The companies that handle failures well aren’t lucky — they planned for them while everything was still fine.