System Failure Case Studies Start a conversation
Twelve cases, one way of reading them

Plants do not fail differently from datacentres.

An offshore platform, a refinery, a nuclear station, three power grids and three datacentres — read with the same question: which function was meant to protect, why did it not, and why did nobody notice beforehand. The answers rhyme across industries far more than anyone involved would like.

Every analysis is here as a PDF to take away — no sign-up, nothing asked in return.

Fukushima Daiichi: four emergency diesels, one location — redundancy is not independence.

What is in the collection

12

Cases

Each with a written analysis and an infographic that puts the sequence on a single sheet.

1988–2025

Time span

From Piper Alpha to the Iberian Peninsula blackout — very nearly four decades.

3

Domains

Process and offshore, power grids, IT infrastructure. The patterns travel between them.

The cases

Grouped by domain, then by year. Clicking the image opens the infographic; the link below opens the full analysis.

Process plants, offshore and nuclear6 cases

1988North Sea, Scotland

Piper Alpha

An offshore platform loses 167 people because a pump is restarted whose safety valve had been removed. The permit existed — the night shift just never saw it.

When a shift handover is the only carrier of safety-critical information, the handover is a single point of failure.

Read the analysis (PDF, 3 pages, 10 kB)

2005Refinery, Texas

BP Texas City

Starting up an isomerisation unit, hydrocarbon overflows, finds an ignition source and kills 15 people. The instrumentation did not show the level that mattered.

An instrument that does not cover the critical range is not an instrument. It is reassurance.

Read the analysis (PDF, 12 pages, 1.4 MB)

2005Fuel depot, Hertfordshire

Buncefield

Tank 912 fills overnight. The level gauge sits still, the independent high-level switch never trips, and by morning a vapour cloud ignites that wakes half of Hemel Hempstead.

The second layer of protection existed, but nobody ever checked that it still tripped. A hidden function nobody looks for is not a function.

Read the analysis (PDF, 12 pages, 1.1 MB)

2010Gulf of Mexico

Deepwater Horizon

The cement job does not seal, the pressure test is read the wrong way, and the blowout preventer closes but does not seal. Eleven people die; three barriers fail in sequence.

Barriers that fail in sequence were never independent. All three rested on the same assumption about the state of the well.

Read the analysis (PDF, 13 pages, 1.4 MB)

2010Refinery, Washington State

Tesoro Anacortes

A heat exchanger ruptures during start-up. The cause had grown for years: hydrogen attacking the steel from within, decarburising it with nothing visible from outside.

A damage mechanism with no detectable warning cannot be managed by inspection. Only the operating limit carries it.

Read the analysis (PDF, 12 pages, 1.3 MB)

2011Nuclear power station, Japan

Fukushima Daiichi

The earthquake cuts the plant from the grid, the emergency diesels take over — and an hour later they are under water, sited low together with the switchgear. From then on the decay heat could not be removed.

Redundancy is not independence. Four diesels in one place are, against a flood, a single diesel.

Read the analysis (PDF, 13 pages, 1.6 MB)

Power grids3 cases

2003United States and Canada

Northeast Blackout

A line touches a tree, the control room alarm system has been dead for hours, and nobody notices the display is frozen. Some 55 million people lose power.

A monitoring system whose own failure is not monitored blinds the operation without anyone noticing the blindness.

Read the analysis (PDF, 13 pages, 1.3 MB)

2021Power grid, Texas

Texas, Winter Storm Uri

The freeze takes generating plants down — and with them the gas production that feeds them. Both sides failed for the same reason, and each had assumed the other as given.

Two systems that count each other as backup have no backup at all when one cause reaches both.

Read the analysis (PDF, 13 pages, 1.2 MB)

2025Spain and Portugal

Iberian Peninsula

Within seconds the peninsula loses its grid. The most recent case in this collection and the only one still under investigation — described here with deliberate restraint.

The faster a grid collapses, the less the operator decides and the more the design of the protection scheme does.

Read the analysis (PDF, 13 pages, 1.3 MB)

IT infrastructure3 cases

2012Microsoft, worldwide

Azure Leap Day

On 29 February a certificate is given an expiry date that does not exist. The calculation fails, nodes restart, and the self-healing mechanism carries the fault across the datacentre.

A date is an operating condition. One you meet every four years is one you have never tested.

Read the analysis (PDF, 12 pages, 1.3 MB)

2017AWS, Northern Virginia

Amazon S3

A typo in a maintenance command removes more servers than intended. Recovery takes hours because nobody had exercised a restart at that scale in years.

A restart that has never been exercised is not a plan but a hope, and its duration stays unknown until the day it matters.

Read the analysis (PDF, 18 pages, 1.9 MB)

2017Datacentre, London

British Airways

It is not the power failure that does the damage but the uncontrolled restoration. On a bank holiday weekend the airline's operation stops worldwide.

The path back to normal operation is part of the design. Plan only for the failure and you have planned for half the event.

Infographic only — the written analysis is still missing

Where this material comes from. The analyses and infographics were produced with Google NotebookLM from the public investigation reports — the Cullen Inquiry, the U.S. Chemical Safety Board, the Buncefield Major Incident Investigation Board, operator post-mortems and grid operator reports. They were reviewed for accuracy afterwards. The NotebookLM mark on the slides is deliberately left where it belongs.

For load-bearing figures, timings and quotations the investigation report governs, not this summary. For the Iberian Peninsula blackout the final report is not yet published.

These case studies are teaching material. They replace neither an investigation nor the judgement of people who know the asset in question. Several of these cases cost lives — they are treated here as lessons, not as anecdotes.

Start a conversation

If you thought about your own plant while reading, that is the usual moment. Write briefly about what stayed with you.

  • The cases as the opening of a workshop — they start conversations a spreadsheet does not.
  • Matched to the RCM course and the course handout, also available here.
  • An answer from someone who knows the method, not from sales.

No newsletter. Never shared with third parties.