The outage I caused on purpose

#

At one in the morning I stopped my Nextcloud stack. Deliberately — 200 GB of photos were moving into proper ZFS datasets so they could finally live under snapshots and replication. Planned downtime, one user affected (me), nothing dramatic.

Then I sat back and watched, because this was also something else: a free, live-fire test of every monitoring system I run. Nobody was told. Nothing was silenced. Whatever happened next would be the truth about my observability.

What happened next

#

Within a minute, ethica.no’s container monitoring saw the nc-app container exit. The public status page flipped to “Some systems are down” with Nextcloud offline — and the service desk opened an incident ticket on its own, with the alert mail threading straight into it. The full ITSM loop, fired by a docker ps nobody was watching.

On this site’s front page, the Homelab Status widget — fed by a tiny collector that pushes snapshots of container state — switched to “1 service down · Nextcloud: exited”. Anyone visiting the blog at 2 AM got the honest version of my basement.

And in snapshoot, the layer underneath told its part of the story: the new datasets appeared on the dashboard fifteen minutes after I created them, correctly yellow — “no snapshots reported”. When the first replication of the new data started, the pair auto-linked mid-transfer and the phone buzzed: replica has no snapshots — true, because the transfer wasn’t done yet.

Three systems. Three different layers — service, container, data. Zero shared code, zero shared config, no system aware the others exist.

They all agreed.

Why agreement matters

#

Any single monitor can lie to you. A green dashboard might mean “everything works” — or “the checker is broken”, “the alert rule was never wired up”, “someone silenced it in 2024”. You can’t tell from inside that one system. This is the monitoring version of a single witness: believable, but uncorroborated.

Independent witnesses change that. My container monitor, my front-page widget and my snapshot guardian observe the world through different mechanisms, on different hosts, written at different times. When they agree — down is down, up is up — each one corroborates the others. And when they ever disagree, that disagreement is itself the alert: one of my witnesses has gone unreliable, and I know which layer to go look at.

I didn’t design this as an architecture. It accumulated, one small honest tool at a time. But the property it produces — cross-checkable truth — is exactly what you want and exactly what a single pane of glass can’t give you.

Planned downtime is a free fire drill

#

Here’s the habit I’m taking away: every piece of planned maintenance is a no-cost test of your entire alerting chain. The stack was down anyway — the only question was whether my systems would notice, in order, at the right layers, all the way to my pocket. They did: status page, auto-ticket, front-page widget, push notification. If any link had stayed quiet, I’d have found a dead alert path for free, at 1 AM, with zero stakes — instead of during a real incident, with all of them.

Most people find out their alerting is broken during the outage that mattered. You can find out during one that didn’t.

The morning after

#

When the migration finished and the stack came back, the whole thing unwound by itself: containers healthy, status page green, ticket resolved, widget back to “All Systems Operational”, datasets green with their first snapshots — and eventually, a restore drill proving the photos are not just copied but readable.

Total human effort on the monitoring side, from outage to all-clear: zero. That’s the standard now.

How many independent witnesses does your infrastructure have — and when did they last agree on something while you watched? If you can’t remember, your next planned maintenance window is a free chance to find out. You know where to find me.