Fractal Techware

Alert runbooks /

NodeTemperatureCritical

A CPU, disk, or board sensor reports that it has crossed its own critical temperature threshold.

   
Severity warning
Source node_exporter 1.x, hwmon collector (bare metal; most cloud VMs expose no sensors)
Key metrics node_hwmon_temp_crit_alarm_celsius, node_hwmon_temp_celsius, node_hwmon_temp_crit_celsius (labels chip, sensor)

What it means

The hwmon collector reads sensors from /sys/class/hwmon. Many drivers expose a critical alarm flag that the chip itself sets when the reading passes the manufacturer’s critical limit. The alert fires when that flag has stayed raised for a few minutes. Because the threshold comes from the hardware, you do not need to guess what “too hot” means for each part.

Expect the CPU to throttle, which shows up as unexplained slowness. If the temperature keeps rising, the firmware will power the machine off to protect it, and sustained heat shortens the life of disks and capacitors.

Common causes

First checks

  1. See every sensor in alarm, then compare readings with their limits:
    node_hwmon_temp_crit_alarm_celsius == 1
    
    node_hwmon_temp_celsius * on(instance, chip, sensor) group_left() (node_hwmon_temp_crit_alarm_celsius == 1)
    
  2. Map cryptic chip labels to driver names:
    node_hwmon_chip_names
    
  3. Read the sensors directly on the host (lm-sensors):
    sensors
    

    On servers with a BMC, sudo ipmitool sdr type Temperature and sudo ipmitool sdr type Fan show chassis sensors and fan speeds.

  4. Check whether the CPU is already throttling:
    rate(node_cpu_core_throttles_total[5m]) > 0
    
    dmesg -T | grep -iE "temperature above threshold|throttl"
    
  5. Check neighbouring hosts in the same rack; if they are all warm, the problem is the room, not the server.

Fixing it

Reduce heat first: drain or migrate workloads off the host. Then fix the cause on site: replace failed fans, clear airflow, raise the fan profile in the BMC, or escalate a cooling failure to facilities. If a single sensor alarms with a plausible reading, check for a firmware or kernel driver update before ignoring it.