NodeCPUHighIOWait

The host’s CPUs have spent a large share of their time idle but blocked on I/O for a sustained period, which means storage (or a network filesystem) is the bottleneck.


Severity	warning
Source	node_exporter 1.x, `cpu` collector
Key metric	`node_cpu_seconds_total{mode="iowait"}`

What it means

iowait is time a CPU had nothing else to run while at least one task was waiting for I/O to complete. It is not CPU work: the machine is waiting on disks. The alert fires when the average iowait share across all CPUs stays high for a long stretch, filtering out short bursts like a nightly backup.

Users feel this as slow requests, database latency, stalled deploys and a rising load average, even though CPU “usage” looks moderate.

Common causes

A disk at its IOPS or throughput limit (cloud volumes with burst credits exhausted are a classic).
A heavy batch job: backup, rsync, compaction, reindexing, log shipping.
Memory pressure forcing constant page cache eviction or swapping.
A slow or unhealthy NFS/iSCSI backend.
A degraded RAID array rebuilding.

First checks

See how iowait is spread over time on this host:

sum by (instance) (rate(node_cpu_seconds_total{mode="iowait", instance="<instance>"}[5m]))

Find the busiest devices:

topk(5, rate(node_disk_io_time_seconds_total{instance="<instance>"}[5m]))

Confirm on the host and watch await, aqu-sz and %util per device:
```
iostat -xz 2 5
```

Find the processes doing the I/O:

sudo iotop -oPa -d 2
# or, without iotop:
pidstat -d 2 5

Rule out memory pressure and swap:

vmstat 2 5   # watch si/so and wa columns

Fixing it

Throttle or reschedule the heavy job (ionice -c3 -p <pid> lowers its I/O priority). If the volume is simply too small for the workload, raise its provisioned IOPS/throughput or move hot data to faster storage. If swapping shows up, fix the memory problem first. For network storage, check the server and the path to it.

NodeDiskIOSaturation: pinpoints which device has a growing queue.
NodeLoadHigh: tasks blocked on I/O count toward load average.
NodeMemoryMajorPagesFaults: memory pressure that turns into disk reads.
NodeHighCPUUsage: distinguishes real CPU saturation from waiting.