The outage I caused on purpose

At one in the morning I stopped my Nextcloud stack. Deliberately — 200 GB of photos were moving into proper ZFS datasets so they could finally live under snapshots and replication. Planned downtime, one user affected (me), nothing dramatic.

Then I sat back and watched, because this was also something else: a free, live-fire test of every monitoring system I run. Nobody was told. Nothing was silenced. Whatever happened next would be the truth about my observability.

What happened next

Within a minute, ethica.no’s container monitoring saw the nc-app container exit. The public status page flipped to “Some systems are down” with Nextcloud offline — and the service desk opened an incident ticket on its own, with the alert mail threading straight into it. The full ITSM loop, fired by a docker ps nobody was watching.

On this site’s front page, the Homelab Status widget — fed by a tiny collector that pushes snapshots of container state — switched to “1 service down · Nextcloud: exited”. Anyone visiting the blog at 2 AM got the honest version of my basement.

And in snapshoot, the layer underneath told its part of the story: the new datasets appeared on the dashboard fifteen minutes after I created them, correctly yellow — “no snapshots reported”. When the first replication of the new data started, the pair auto-linked mid-transfer and the phone buzzed: replica has no snapshots — true, because the transfer wasn’t done yet.

Three systems. Three different layers — service, container, data. Zero shared code, zero shared config, no system aware the others exist.

They all agreed.

Why agreement matters

Any single monitor can lie to you. A green dashboard might mean “everything works” — or “the checker is broken”, “the alert rule was never wired up”, “someone silenced it in 2024”. You can’t tell from inside that one system. This is the monitoring version of a single witness: believable, but uncorroborated.

Independent witnesses change that. My container monitor, my front-page widget and my snapshot guardian observe the world through different mechanisms, on different hosts, written at different times. When they agree — down is down, up is up — each one corroborates the others. And when they ever disagree, that disagreement is itself the alert: one of my witnesses has gone unreliable, and I know which layer to go look at.

I didn’t design this as an architecture. It accumulated, one small honest tool at a time. But the property it produces — cross-checkable truth — is exactly what you want and exactly what a single pane of glass can’t give you.

Planned downtime is a free fire drill

Here’s the habit I’m taking away: every piece of planned maintenance is a no-cost test of your entire alerting chain. The stack was down anyway — the only question was whether my systems would notice, in order, at the right layers, all the way to my pocket. They did: status page, auto-ticket, front-page widget, push notification. If any link had stayed quiet, I’d have found a dead alert path for free, at 1 AM, with zero stakes — instead of during a real incident, with all of them.

Most people find out their alerting is broken during the outage that mattered. You can find out during one that didn’t.

The morning after

When the migration finished and the stack came back, the whole thing unwound by itself: containers healthy, status page green, ticket resolved, widget back to “All Systems Operational”, datasets green with their first snapshots — and eventually, a restore drill proving the photos are not just copied but readable.

Total human effort on the monitoring side, from outage to all-clear: zero. That’s the standard now.

How many independent witnesses does your infrastructure have — and when did they last agree on something while you watched? If you can’t remember, your next planned maintenance window is a free chance to find out. You know where to find me.