Overview

#

A few days ago I wrote about a VM that occasionally dropped into an initramfs shell due to filesystem corruption.

At the time, I had more questions than answers.

Since then, I’ve spent several evenings chasing clues through ZFS, SMART data, kernel logs, storage controllers, and hardware diagnostics.

The strange part?

Everything looks healthy.

Recap

#

The affected VM hosts several game servers through AMP.

When the issue appears, the VM eventually becomes unavailable and Ubuntu refuses to boot normally.

Instead, it drops directly into:

(initramfs)

A simple filesystem repair restores normal operation:

fsck -f -y /dev/sda1

The repair succeeds.

The VM boots.

Everything works again.

Until the next incident.

First Theory: Storage Corruption

#

My first suspicion was storage.

The VM runs on a ZFS-backed mirror of two Intel enterprise SSDs.

Checking the pools:

zpool status -v

Output:

pool: rpool
state: ONLINE

errors: No known data errors

pool: rust
state: ONLINE

errors: No known data errors

Not exactly the smoking gun I was hoping for.

No checksum errors.

No read errors.

No write errors.

No repaired data.

Looking at the VM Configuration

#

The affected VM is configured as:

qm config 120

Relevant output:

name: Game
memory: 34096
cores: 4

scsi0: local-zfs:vm-120-disk-0,size=150G
scsihw: virtio-scsi-pci

The virtual disk lives on the SSD mirror (local-zfs).

Nothing unusual here.

SMART Diagnostics

#

Next stop was the SSDs.

smartctl -x /dev/disk/by-id/ata-INTEL_SSDSC2BX800G4R_BTHC64230CAL800NGN

Highlights:

SMART overall-health self-assessment test result: PASSED

Reallocated_Sector_Ct: 0
Offline_Uncorrectable: 0
CRC_Error_Count: 0

No Errors Logged

The drive has over 71,000 power-on hours, but still reports healthy status.

Even more interesting:

Percentage Used Endurance Indicator: 2

For an SSD this old, that’s impressive.

Again, no obvious evidence of hardware failure.

Memory?

#

The next theory was memory corruption.

Checking kernel logs:

journalctl -k | grep -Ei "mce|edac|ecc|memory"

Result:

EDAC MC0
EDAC MC1
EDAC MC2
EDAC MC3

No machine check exceptions.

No ECC events.

No hardware memory errors.

The memory subsystem appears healthy.

Looking for I/O Problems

#

If storage was failing, I expected to find timeout or I/O related messages.

So I searched:

journalctl -k | grep -Ei "blocked|hung|timeout|reset|i/o"

Result:

megaraid_sas ... Online Controller Reset(OCR): Enabled
power_meter ACPI000D:00: Waiting for ACPI IPMI timeout

Nothing that clearly points to storage instability.

No failed writes.

No timeout storms.

No controller crashes.

The RAID Controller Rabbit Hole

#

While inspecting the system I discovered:

lspci | grep -Ei "raid|sas|lsi|broadcom"

Output:

02:00.0 RAID bus controller:
Broadcom / LSI MegaRAID SAS-3 3108 [Invader]

Interesting.

For a moment I thought I had found the culprit.

However, checking disk visibility:

lsblk -o NAME,SIZE,TYPE,MODEL

Showed:

sda INTEL SSDSC2BX800G4R
sdb INTEL SSDSC2BX800G4R

sdc AL14SEB120N
sdd ST1200MM0007
sde ST1200MM0007
sdf HUC101812CSS204

The disks are exposed individually.

Nothing suggests hidden RAID virtual disks confusing ZFS.

Another promising lead eliminated.

The Detail That Doesn’t Fit

#

One incident changed how I think about this problem.

Players continued playing on the game servers.

Hours later, several hours of progress had disappeared.

The applications appeared alive.

The operating system did not.

That behaviour doesn’t match a simple crash.

It suggests writes may have stopped reaching persistent storage while processes continued running.

Which raises another possibility:

What if the filesystem became read-only?

Current Theory

#

After checking:

• ZFS
• SMART data
• Memory diagnostics
• Kernel logs
• Storage controller information

I am less convinced that the host is responsible.

Ironically, the deeper I investigate the hardware, the healthier it appears.

My current theory is that something inside the guest operating system is triggering filesystem issues which later force the system into recovery mode.

The evidence is still incomplete.

But at the moment it explains more observations than any storage-related theory.

Lessons Learned

#

One of the most frustrating parts of troubleshooting is when every diagnostic comes back clean.

A broken disk is easy.

A failed memory module is easy.

A healthy system with an unhealthy workload is much harder.

The filesystem can be repaired.

The VM can be restored.

Finding the reason it happened in the first place is proving far more difficult.

And that’s exactly why this post exists.

The investigation continues.

Or, as always in a homelab:

Something broke while I wasn’t paying attention.