Prometheus alert runbooks

Mid-incident and staring at an unfamiliar alert name? Each page below explains what a common Prometheus alert means in plain words, what usually causes it, and the first commands to run (kubectl, PromQL, shell) to find out what is actually wrong.

The alerts come from the Prometheus Alert Rules & Runbook Pack: 179 alerts across 20 domains, every one covered by promtool unit tests. 12 of them, with their rules, tests and full runbooks, are free under MIT in prometheus-alert-rules on GitHub; those pages are marked free rule.

Kubernetes workloads

• KubePodCrashLooping: Pod container is crash looping. (warning, free rule)
• KubePodNotReady: Pod has been in a non-ready state. (warning, free rule)
• KubeContainerOOMKilled: Container was OOM killed. (warning, free rule)
• KubeImagePullBackOff: Container image cannot be pulled. (warning, free rule)
• KubeContainerWaiting: Container has been waiting to start for a long time. (warning)
• KubeDeploymentReplicasMismatch: Deployment has fewer available replicas than desired. (warning, free rule)
• KubeDeploymentRolloutStuck: Deployment rollout is not progressing. (warning, free rule)
• KubeDeploymentGenerationMismatch: Deployment generation mismatch. (warning)
• KubeStatefulSetReplicasMismatch: StatefulSet has not matched the expected number of ready replicas. (warning)
• KubeStatefulSetGenerationMismatch: StatefulSet generation mismatch. (warning)
• KubeDaemonSetRolloutStuck: DaemonSet rollout is stuck. (warning)
• KubeDaemonSetNotScheduled: DaemonSet pods are not scheduled. (warning)
• KubeDaemonSetMisScheduled: DaemonSet pods are running where they are not supposed to run. (warning)

Kubernetes nodes & capacity

• KubeNodeNotReady: Node is not ready. (warning, free rule)
• KubeNodeUnreachable: Node is unreachable. (warning)
• KubeNodePressure: Node has an active pressure condition. (warning)
• KubeNodeReadinessFlapping: Node readiness is flapping. (warning)
• KubeletTooManyPods: Node is running at its pod capacity. (warning)
• KubeNodeCordoned: Node has been cordoned for a long time. (info)
• KubeClusterCPURequestsHigh: Cluster CPU requests are close to allocatable capacity. (warning)
• KubeClusterMemoryRequestsHigh: Cluster memory requests are close to allocatable capacity. (warning)

Hosts (node_exporter)

• NodeExporterDown: Host or node_exporter is down. (critical, free rule)
• NodeFilesystemSpaceFillingUp: Filesystem is predicted to run out of space within 24 hours. (warning / critical, free rule)
• NodeFilesystemAlmostOutOfSpace: Filesystem has less than 10% space left. (warning / critical, free rule)
• NodeFilesystemAlmostOutOfFiles: Filesystem has less than 10% inodes left. (warning / critical)
• NodeFilesystemDeviceError: Filesystem cannot be read by node_exporter. (warning)
• NodeHighCPUUsage: Host CPU usage is above 90%. (warning, free rule)
• NodeCPUHighIOWait: Host CPUs are spending a lot of time waiting for I/O. (warning)
• NodeLoadHigh: Host load average is high relative to its CPU count. (warning)
• NodeMemoryHighUtilization: Host memory utilisation is above 90%. (warning, free rule)
• NodeMemoryMajorPagesFaults: Host is under memory pressure (major page faults). (warning)
• NodeOOMKillDetected: The kernel OOM killer terminated a process. (warning)
• NodeDiskIOSaturation: Disk I/O queue is saturated. (warning)
• NodeNetworkReceiveErrs: Network interface is reporting receive errors. (warning)
• NodeNetworkTransmitErrs: Network interface is reporting transmit errors. (warning)
• NodeNetworkInterfaceFlapping: Network interface is flapping. (warning)
• NodeClockNotSynchronising: Host clock is not synchronising. (warning)
• NodeClockSkewDetected: Host clock is skewed. (warning)
• NodeSystemdServiceFailed: A systemd service has failed. (warning)
• NodeRAIDDegraded: Software RAID array is degraded. (critical)
• NodeRAIDDiskFailure: Disk in a software RAID array has failed. (warning)
• NodeTextFileCollectorError: node_exporter textfile collector failed. (warning)
• NodeRebootDetected: Host rebooted. (info)
• NodeFileDescriptorLimit: Host is running out of file descriptors. (warning)
• NodeConntrackLimit: Conntrack table is almost full. (warning)
• NodeTemperatureCritical: Hardware sensor reports a critical temperature. (warning)

Kubernetes persistent volumes

• KubePersistentVolumeFillingUp: PersistentVolume is almost full. (warning / critical)
• KubePersistentVolumeInodesFillingUp: PersistentVolume is running out of inodes. (critical)
• KubePersistentVolumeErrors: PersistentVolume is in a failed or pending state. (critical)
• KubePersistentVolumeClaimPending: PersistentVolumeClaim is not bound. (warning)
• KubePersistentVolumeClaimLost: PersistentVolumeClaim lost its volume. (critical)

Kubernetes Jobs & CronJobs

• KubeJobFailed: Job failed to complete. (warning)
• KubeJobNotCompleted: Job is taking too long to complete. (warning)
• KubeCronJobMissedSchedule: CronJob did not run on schedule. (warning)

Kubernetes autoscaling (HPA)

• KubeHpaReplicasMismatch: HPA has not reached its desired number of replicas. (warning)
• KubeHpaMaxedOut: HPA is running at maximum replicas. (warning)
• KubeHpaUnableToScale: HPA is unable to scale its target. (warning)
• KubeHpaMetricsUnavailable: HPA cannot read the metrics it scales on. (warning)

Kubernetes quotas, limits & disruption budgets

• KubeQuotaAlmostFull: Namespace quota is almost full. (info)
• KubeQuotaFullyUsed: Namespace quota is fully used. (info)
• KubeQuotaExceeded: Namespace quota is exceeded. (warning)
• CPUThrottlingHigh: Container is heavily CPU throttled. (info)
• KubeContainerMemoryNearLimit: Container memory usage is close to its limit. (warning)
• KubePodDisruptionBudgetViolated: PodDisruptionBudget is violated. (warning)
• KubePodDisruptionBudgetBlocksEviction: PodDisruptionBudget allows no disruptions. (info)

Kubernetes control plane & kubelet

• KubeAPIDown: Kubernetes API server is unreachable. (critical)
• KubeAPIErrorsHigh: Kubernetes API server is returning many 5xx errors. (warning / critical)
• KubeAPILatencyHigh: Kubernetes API server is slow. (warning)
• KubeAPITerminatedRequests: Kubernetes API server is terminating requests. (warning)
• KubeletDown: No kubelet can be scraped. (critical)
• KubeSchedulerDown: kube-scheduler has disappeared. (critical)
• KubeControllerManagerDown: kube-controller-manager has disappeared. (critical)
• KubeVersionMismatch: Different Kubernetes versions are running. (warning)
• KubeAggregatedAPIDown: An aggregated API is down. (warning)
• KubeClientErrors: Kubernetes API client is experiencing errors. (warning)
• KubeletPlegDurationHigh: Kubelet pod lifecycle event generator is slow. (warning)
• KubeletPodStartUpLatencyHigh: Kubelet pod startup latency is high. (warning)

Kubernetes certificates (kubelet & cert-manager)

• KubeletClientCertificateExpiration: Kubelet client certificate is about to expire. (warning / critical)
• KubeletServerCertificateExpiration: Kubelet serving certificate is about to expire. (warning / critical)
• KubeletClientCertificateRenewalErrors: Kubelet fails to renew its client certificate. (warning)
• KubeletServerCertificateRenewalErrors: Kubelet fails to renew its serving certificate. (warning)
• CertManagerCertificateExpiringSoon: cert-manager certificate expires soon and has not been renewed. (warning / critical)
• CertManagerCertificateNotReady: cert-manager certificate is not ready. (warning)

Prometheus self-monitoring

• Watchdog: Alerting pipeline heartbeat (always firing). (info)
• TargetDown: Some scrape targets are down. (warning)
• PrometheusConfigReloadFailed: Prometheus configuration reload failed. (warning)
• PrometheusNotConnectedToAlertmanagers: Prometheus is not connected to any Alertmanager. (warning)
• PrometheusErrorSendingAlertsToAlertmanager: Prometheus has errors sending alerts to an Alertmanager. (warning)
• PrometheusNotificationQueueRunningFull: Prometheus alert notification queue predicted to run full. (warning)
• PrometheusTSDBReloadsFailing: Prometheus has issues reloading blocks from disk. (warning)
• PrometheusTSDBCompactionsFailing: Prometheus has issues compacting blocks. (warning)
• PrometheusNotIngestingSamples: Prometheus is not ingesting samples. (warning)
• PrometheusDuplicateTimestamps: Prometheus is dropping samples with duplicate timestamps. (warning)
• PrometheusOutOfOrderTimestamps: Prometheus drops samples with out-of-order timestamps. (warning)
• PrometheusRemoteStorageFailures: Prometheus fails to send samples to remote storage. (critical)
• PrometheusRemoteWriteBehind: Prometheus remote write is falling behind. (critical)
• PrometheusRuleFailures: Prometheus is failing rule evaluations. (critical)
• PrometheusMissingRuleEvaluations: Prometheus is missing rule evaluations due to slow rule group evaluation. (warning)
• PrometheusTargetLimitHit: Prometheus has dropped targets because some scrape configs exceeded the targets limit. (warning)
• PrometheusLabelLimitHit: Prometheus has dropped targets because some scrape configs exceeded the label limits. (warning)
• PrometheusScrapeSampleLimitHit: Prometheus rejects scrapes that exceed the sample limit. (warning)
• PrometheusSDRefreshFailure: Prometheus service discovery is failing. (warning)
• PrometheusHighQueryLoad: Prometheus is reaching its maximum query capacity. (warning)

Alertmanager self-monitoring

• AlertmanagerFailedReload: Reloading an Alertmanager configuration has failed. (critical)
• AlertmanagerMembersInconsistent: A member of an Alertmanager cluster has not found all other cluster members. (critical)
• AlertmanagerFailedToSendAlerts: An Alertmanager instance failed to send notifications. (warning)
• AlertmanagerConfigInconsistent: Alertmanager instances within the same cluster have different configurations. (critical)
• AlertmanagerClusterDown: Half or more of the Alertmanager instances within the same cluster are down. (critical)
• AlertmanagerClusterCrashlooping: Half or more of the Alertmanager instances within the same cluster are crashlooping. (critical)

Endpoint probes (blackbox_exporter)

• BlackboxProbeFailed: Endpoint probe is failing. (critical)
• BlackboxProbeFlapping: Endpoint probe is flapping. (warning)
• BlackboxSlowProbe: Endpoint probe is slow. (warning)
• BlackboxSslCertificateWillExpireSoon: TLS certificate of a probed endpoint expires soon. (warning / critical)
• BlackboxDnsLookupSlow: DNS resolution for a probed endpoint is slow. (warning)
• BlackboxExporterProbeScrapeFailed: Prometheus cannot reach blackbox_exporter for a probe. (warning)

PostgreSQL (postgres_exporter)

• PostgresqlDown: PostgreSQL is down. (critical)
• PostgresqlTooManyConnections: PostgreSQL is close to its connection limit. (warning)
• PostgresqlReplicationLagHigh: PostgreSQL replica is lagging. (warning)
• PostgresqlDeadlocks: PostgreSQL is detecting deadlocks. (warning)
• PostgresqlHighRollbackRate: PostgreSQL has a high transaction rollback rate. (warning)
• PostgresqlLongRunningTransaction: PostgreSQL has a long-running transaction. (warning)
• PostgresqlHighDeadTuples: PostgreSQL table has many dead tuples. (warning)
• PostgresqlCacheHitRatioLow: PostgreSQL buffer cache hit ratio is low. (info)
• PostgresqlExporterScrapeError: postgres_exporter reports scrape errors. (warning)
• PostgresqlRestarted: PostgreSQL restarted. (info)

Redis (redis_exporter)

• RedisDown: Redis is down. (critical)
• RedisMissingMaster: Redis has no master. (critical)
• RedisReplicaLinkDown: Redis replica lost its link to the master. (critical)
• RedisReplicasDisconnected: Redis master lost replicas. (warning)
• RedisMemoryHigh: Redis is close to its maxmemory limit. (warning)
• RedisTooManyConnections: Redis is close to its client connection limit. (warning)
• RedisRejectedConnections: Redis is rejecting connections. (warning)
• RedisRdbLastSaveFailed: Redis RDB snapshot failed. (warning)
• RedisKeyEvictions: Redis is evicting keys. (info)

NGINX Ingress Controller

• NginxIngressHighHttp5xxErrorRate: Ingress is returning many 5xx responses. (warning / critical)
• NginxIngressHighHttp4xxErrorRate: Ingress is returning many 4xx responses. (info)
• NginxIngressHighLatency: Ingress p95 latency is high. (warning)
• NginxIngressConfigReloadFailed: NGINX Ingress configuration reload failed. (warning)
• NginxIngressCertificateExpiring: Certificate served by the ingress controller expires soon. (warning / critical)

Apache Kafka (kafka_exporter)

• KafkaBrokersDown: Kafka cluster has fewer brokers than expected. (critical)
• KafkaUnderReplicatedPartitions: Kafka topic has under-replicated partitions. (warning)
• KafkaTopicPartitionNoLeader: Kafka partitions have no leader. (critical)
• KafkaConsumerGroupLagHigh: Kafka consumer group lag is high. (warning)
• KafkaConsumerGroupLagGrowing: Kafka consumer group lag keeps growing. (warning)
• KafkaConsumerGroupInactive: Kafka consumer group has lag but no members. (warning)

Grafana Loki

• LokiRequestErrors: Loki is returning many 5xx errors. (critical)
• LokiRequestPanics: Loki is panicking. (critical)
• LokiRequestLatency: Loki requests are slow. (warning)
• LokiDiscardedSamples: Loki is discarding log lines. (warning)
• LokiCompactorHasNotSuccessfullyRunCompaction: Loki compaction has not run recently. (warning)

CoreDNS

• CoreDNSDown: CoreDNS has disappeared from Prometheus target discovery. (critical)
• CoreDNSLatencyHigh: CoreDNS is responding slowly. (warning)
• CoreDNSErrorsHigh: CoreDNS is returning SERVFAIL for many requests. (warning / critical)
• CoreDNSForwardLatencyHigh: CoreDNS upstream forwarding is slow. (warning)
• CoreDNSForwardHealthcheckFailures: CoreDNS health checks to an upstream resolver are failing. (warning)
• CoreDNSPanics: CoreDNS is panicking. (critical)

etcd

• EtcdInsufficientMembers: etcd cluster has insufficient members for quorum. (critical)
• EtcdMembersDown: etcd cluster members are down. (warning)
• EtcdNoLeader: etcd member has no leader. (critical)
• EtcdHighNumberOfLeaderChanges: etcd leader changes too often. (warning)
• EtcdHighFsyncDurations: etcd WAL fsync is slow. (warning)
• EtcdHighCommitDurations: etcd backend commits are slow. (warning)
• EtcdDatabaseQuotaLowSpace: etcd database is close to its size quota. (warning / critical)
• EtcdHighNumberOfFailedGRPCRequests: etcd is failing many gRPC requests. (warning)

HTTP availability SLO (multi-window, multi-burn-rate)

• ErrorBudgetBurn: Service is burning its error budget very fast. (warning / critical)