Fractal Techware

Alert runbooks /

RedisMissingMaster

None of the Redis instances in a group currently report themselves as master, so there is nowhere for writes to go.

   
Severity critical
Source oliver006/redis_exporter 1.50+
Key metric redis_instance_info (label role = master or slave)

What it means

Every scraped instance exposes its role. The alert groups instances by cluster and job and fires when that group has no master for a couple of minutes, including the case where the master’s series has vanished entirely.

Replicas are read-only by default, so any write returns READONLY You can't write against a read only replica. Reads may continue from replicas, but they are serving data that will not change until a master returns.

Common causes

First checks

  1. See what each exporter currently reports:
    redis_instance_info
    

    Check whether a master series is missing, or a former master now shows role="slave".

  2. Ask each instance directly:
    redis-cli -h <host> INFO replication | grep -E '^(role|master_host|master_link_status|connected_slaves)'
    
  3. If you use Sentinel, ask it what it believes:
    redis-cli -p 26379 SENTINEL get-master-addr-by-name <master-name>
    redis-cli -p 26379 SENTINEL ckquorum <master-name>
    redis-cli -p 26379 SENTINEL replicas <master-name>
    
  4. Check the Sentinel logs for +sdown, +odown, +failover-abort-* or -failover-abort-no-good-slave events.
  5. Confirm the exporter for the old master is up: up{job="<redis-exporter-job>"}.

Fixing it

If the old master is healthy and just unmonitored, fix its exporter. If it is gone and Sentinel is stuck, restore Sentinel quorum (bring back the missing Sentinels) or force a promotion with SENTINEL failover <master-name>. Without Sentinel, pick the replica with the highest master_repl_offset and run REPLICAOF NO ONE on it, then repoint the other replicas and the application. Accept that writes after that offset are lost.