Fractal Techware

Alert runbooks /

NodeClockNotSynchronising

The kernel on this host reports that its clock is no longer being disciplined by a time source, so it will slowly drift.

   
Severity warning
Source node_exporter 1.x, timex collector (Linux, enabled by default)
Key metrics node_timex_sync_status, node_timex_maxerror_seconds

What it means

node_exporter reads the kernel’s adjtimex state. node_timex_sync_status is 1 while an NTP daemon (chrony, ntpd, systemd-timesyncd) is actively steering the clock and 0 when it is not. The alert fires when the host has reported “unsynchronised” continuously for several minutes and the kernel’s own maximum error estimate has grown large, which rules out a brief blip during a daemon restart.

Nothing breaks immediately, but the clock is now free-running. Over hours or days that drift turns into TLS validation failures, rejected tokens (Kerberos, JWT, AWS request signing), confusing log timelines and, for distributed databases, consistency problems. Treat it as a chance to fix time before NodeClockSkewDetected fires.

Common causes

First checks

  1. See which hosts are affected:
    node_timex_sync_status == 0
    
  2. Ask the OS what it thinks:
    timedatectl status
    

    Look at “System clock synchronized” and “NTP service”.

  3. If chrony is in use, check that it has a selected source (marked *):
    chronyc tracking
    chronyc sources -v
    

    For systemd-timesyncd use timedatectl timesync-status; for ntpd use ntpq -p.

  4. Confirm the daemon is running and see why it is not syncing:
    systemctl status chronyd systemd-timesyncd ntpd 2>/dev/null
    journalctl -u chronyd -b --no-pager | tail -50
    
  5. Test reachability of a server: chronyc -n sources showing ? for every source usually points to a network or DNS problem.

Fixing it

Run exactly one time daemon, point it at reachable servers (your cloud provider’s endpoint is usually best), open UDP 123 outbound, then restart it. If the clock is already far off, chronyc makestep corrects it immediately instead of slewing slowly. Bake the working configuration into your base image so new hosts do not repeat the problem.