NodeRAIDDegraded

A software RAID array has lost redundancy: it is still serving data, but one more failure could lose it.


Severity	critical
Source	node_exporter 1.x, `mdadm` collector (reads `/proc/mdstat`)
Key metrics	`node_md_disks_required`, `node_md_disks{state="active"}`

What it means

For every md array, node_exporter reports how many member disks the array is designed to have and how many are currently active. The alert fires when the active count stays below the required count for several minutes, meaning a member is missing, failed, or still being rebuilt.

It is critical because a degraded RAID1, RAID5 or RAID10 has no margin left. A second disk error, or a read error hit during rebuild, can take the array and its filesystem down. Performance is usually worse too, since reads must be reconstructed.

Common causes

A disk failed and was kicked out of the array (see NodeRAIDDiskFailure).
A disk disappeared from the bus: loose cable, backplane or controller issue, NVMe dropping off.
The array was assembled at boot without one member, for example after a disk was slow to appear.
A disk was replaced but never added back to the array.
A resync or recovery is in progress after an unclean shutdown.

First checks

See which arrays are short and by how much:

node_md_disks_required - ignoring(state) node_md_disks{state="active"}

Check the kernel’s view, including rebuild progress:
```
cat /proc/mdstat
```
Get the detailed state and which slot is missing or faulty:
```
sudo mdadm --detail /dev/md<N>
```

Check the health of the remaining and missing disks:

sudo smartctl -H -a /dev/<disk>
dmesg -T | grep -iE "md/|ata|nvme|I/O error"

Verify your backups of this host are recent before touching anything.

Fixing it

If a rebuild is already running, let it finish and avoid heavy I/O. If a disk dropped out but SMART is clean, re-add it with mdadm --manage /dev/md<N> --re-add /dev/<part> (or --add). If the disk is bad, remove it (--fail then --remove), replace the hardware, copy the partition table from a healthy member, and --add the new partition. Keep a hot spare on important arrays so recovery starts automatically.

NodeRAIDDiskFailure: a member is explicitly marked failed.
NodeFilesystemDeviceError: the filesystem on the array can no longer be read.
NodeDiskIOSaturation: rebuilds and degraded reads saturate disks.