Fractal Techware

Alert runbooks /

NodeRAIDDiskFailure

The kernel has marked at least one member of a software RAID array as faulty.

   
Severity warning
Source node_exporter 1.x, mdadm collector
Key metric node_md_disks{state="failed"} (labels device, instance)

What it means

node_exporter counts the members of each md array by state (active, failed, spare, and others). This alert fires as soon as the failed count is above zero. The md driver sets a member faulty after I/O errors, or when an administrator fails it by hand.

On its own it is a warning: if a spare took over, or the array has extra redundancy (RAID6, three-way mirror), data is still protected. It becomes urgent when the array is also degraded, so check both.

Common causes

First checks

  1. Find the array and host: 
    node_md_disks{state="failed"} > 0
  2. Identify the faulty member ((F) in mdstat): 
    cat /proc/mdstat
sudo mdadm --detail /dev/md<N>
  3. Look at the disk’s SMART data and error log: 
    sudo smartctl -a /dev/<disk>

    For NVMe, sudo nvme smart-log /dev/nvme<N> gives media errors and wear.

  4. Check kernel messages for the reason it was failed: 
    dmesg -T | grep -iE "raid|md/|I/O error|reset" | tail -40
  5. Map the device to a physical slot before pulling anything: ls -l /dev/disk/by-id/ shows the serial number.

Fixing it

If SMART shows real media errors, replace the disk. Remove it from the array first:

sudo mdadm --manage /dev/md<N> --remove /dev/<failed-part>

After swapping hardware, recreate the partition layout (for GPT, sgdisk --replicate=/dev/<new> /dev/<healthy> then sgdisk -G /dev/<new>), add it with mdadm --manage /dev/md<N> --add /dev/<new-part> and watch /proc/mdstat until recovery completes. If the disk looks healthy and the failure was a cable or controller glitch, fix that cause and re-add the member, but keep an eye on it.