Fractal Techware

Alert runbooks /

KafkaConsumerGroupInactive

A consumer group still has committed offsets and a backlog, but nobody is consuming for it.

   
Severity warning
Source danielqsj/kafka_exporter 1.7+
Key metrics kafka_consumergroup_members, kafka_consumergroup_lag

What it means

Kafka keeps a group’s committed offsets after all its members leave (until offsets.retention.minutes passes). kafka_exporter reports the member count and the lag for that group. The alert fires when a group has had zero members and non-zero lag for a while: messages are arriving for a consumer that no longer exists or is not running.

Either an application is down and its work is piling up, or a group was retired and its offsets are just leftovers.

Common causes

First checks

  1. Confirm the group is empty:
    kafka-consumer-groups.sh --bootstrap-server <broker>:9092 --describe --group <group> --state
    kafka-consumer-groups.sh --bootstrap-server <broker>:9092 --describe --group <group>
    

    State Empty and no CONSUMER-ID confirm it.

  2. Find the owning application and check it is running:
    kubectl get deploy,statefulset -A | grep -i <app>
    kubectl -n <namespace> get pods -l <consumer-selector>
    
  3. Check its logs for join or auth errors:
    kubectl -n <namespace> logs deploy/<consumer> --since=1h | grep -iE 'authoriz|authenticat|group|coordinator'
    
  4. Check whether a new group id replaced it:
    sum by (consumergroup) (kafka_consumergroup_members{consumergroup=~"<app-prefix>.*"})
    

Fixing it

Restore the consumer if it should be running; it resumes from the committed offsets. If the group id changed intentionally, decide whether the new group needs the old offsets and reset it with --reset-offsets before starting. For retired groups, delete them so the alert clears:

kafka-consumer-groups.sh --bootstrap-server <broker>:9092 --delete --group <group>

For periodic batch consumers, exclude the group from this alert.