Alert runbooks /

KubeDeploymentRolloutStuck

Deployment rollout is not progressing. Free rule (MIT): the complete runbook and rule definition are below.

Meaning

The Deployment controller set Progressing=False (reason ProgressDeadlineExceeded) because the rollout made no progress within progressDeadlineSeconds.

Impact

The new version is not fully rolled out. Old and new ReplicaSets may be serving side by side.

Diagnosis

• kubectl -n <namespace> rollout status deployment/<deployment> --timeout=5s
  kubectl -n <namespace> describe deployment <deployment> | sed -n '/Conditions/,/Events/p'
  kubectl -n <namespace> get rs -l <selector>
  

• Inspect the newest ReplicaSet’s pods for readiness or scheduling failures.

Mitigation

• Roll back: kubectl -n <namespace> rollout undo deployment/<deployment>.
• Or fix the underlying pod problem and re-trigger: kubectl -n <namespace> rollout restart deployment/<deployment>.

Escalation

• warning: Notify the owning team’s alert channel. Acknowledge within business hours; create a ticket if it cannot be resolved the same day. Escalate to critical handling if it trends toward the critical condition.

Route to the team that owns the namespace. Platform on-call only takes over when several namespaces are affected at once (likely a node, network or control-plane problem).

Related alerts

• KubePodCrashLooping: Pod container is crash looping.
• KubePodNotReady: Pod has been in a non-ready state.
• KubeContainerOOMKilled: Container was OOM killed.
• KubeImagePullBackOff: Container image cannot be pulled.
• KubeDeploymentReplicasMismatch: Deployment has fewer available replicas than desired.

Rule definition

From rules/kubernetes-workloads.rules.yml in the free repository (MIT). Unit tests for it are in tests/.

- alert: KubeDeploymentRolloutStuck
  expr: kube_deployment_status_condition{job="kube-state-metrics", namespace=~".+", condition="Progressing", status="false"} != 0
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: Deployment rollout is not progressing.
    description: Rollout of deployment {{ $labels.namespace }}/{{ $labels.deployment }} has exceeded its progress deadline and has not progressed for 15m.
    runbook_url: runbooks/kubernetes-workloads/KubeDeploymentRolloutStuck.md