Alert runbooks /

NodeMemoryHighUtilization

Host memory utilisation is above 90%. Free rule (MIT): the complete runbook and rule definition are below.

Meaning

Less than 10% of memory is available (MemAvailable accounts for reclaimable page cache).

Impact

Risk of the kernel OOM killer terminating processes, heavy page-cache eviction and swapping.

Diagnosis

• On the host: free -m, ps aux --sort=-rss | head, smem -tk if installed.

• node_memory_MemAvailable_bytes{instance="<instance>"}
  

• Kubernetes: kubectl top pods -A --sort-by=memory | head.

Mitigation

• Restart or limit the leaking process; set memory limits on containers.
• Add memory (bigger instance) or move workloads away.

Escalation

• warning: Notify the owning team’s alert channel. Acknowledge within business hours; create a ticket if it cannot be resolved the same day. Escalate to critical handling if it trends toward the critical condition.

Owned by whoever operates the host (platform team for Kubernetes nodes, service owner for standalone VMs).

Related alerts

• NodeExporterDown: Host or node_exporter is down.
• NodeFilesystemSpaceFillingUp: Filesystem is predicted to run out of space within 24 hours.
• NodeFilesystemAlmostOutOfSpace: Filesystem has less than 10% space left.
• NodeHighCPUUsage: Host CPU usage is above 90%.
• NodeFilesystemAlmostOutOfFiles: Filesystem has less than 10% inodes left.

Rule definition

From rules/node-exporter.rules.yml in the free repository (MIT). Unit tests for it are in tests/.

- alert: NodeMemoryHighUtilization
  expr: 100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100) > 90
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: Host memory utilisation is above 90%.
    description: Memory utilisation on {{ $labels.instance }} is {{ $value | printf "%.1f" }}%.
    runbook_url: runbooks/node-exporter/NodeMemoryHighUtilization.md