Alert runbooks /

KubePersistentVolumeFillingUp

A PersistentVolumeClaim is running out of free space, and the application writing to it will start failing when it hits zero.

What it means

The kubelet reports capacity and free bytes for every mounted PVC. The alert has two levels:

• warning: free space is already low and the recent trend says the volume will be full within a few days. You have time to plan.
• critical: only a sliver of space is left. Writes may start failing in minutes.

Read-only volumes are excluded. When a volume fills, databases crash or go read-only, queues stop accepting messages and log shippers drop data.

Common causes

• Database growth, WAL or binlog retention, or a replication slot holding old segments.
• Application logs or temp files written to the data volume instead of stdout.
• Retention or compaction stopped (a failed cron, a broken Prometheus/Loki compactor).
• The volume was simply sized too small for normal growth.
• Backups or dumps written locally and never removed.

First checks

1. Find the fullest volumes and how fast they are growing:

   sort_desc(kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes)
   

   deriv(kubelet_volume_stats_used_bytes{namespace="<ns>", persistentvolumeclaim="<pvc>"}[1h])
   

2. Find the pod that mounts the claim:

   kubectl -n <ns> get pods -o json | jq -r '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="<pvc>") | .metadata.name'
   

3. See what is using the space from inside the pod:

   kubectl -n <ns> exec <pod> -- df -h <mount-path>
   kubectl -n <ns> exec <pod> -- du -xh --max-depth=2 <mount-path> | sort -h | tail -20
   

4. Check whether the StorageClass allows online expansion:

   kubectl get storageclass <class> -o jsonpath='{.allowVolumeExpansion}'
   

Fixing it

For a quick win, delete what should not be there (old dumps, rotated logs) or fix the retention job that stopped. If the growth is legitimate, expand the claim when the StorageClass supports it:

kubectl -n <ns> patch pvc <pvc> -p '{"spec":{"resources":{"requests":{"storage":"<new-size>"}}}}'

Watch kubectl -n <ns> describe pvc <pvc> for the resize conditions; some CSI drivers need a pod restart to grow the filesystem. For StatefulSets, also raise the size in the volumeClaimTemplate so new replicas match.

Related alerts

• KubePersistentVolumeInodesFillingUp: the same volume can fail with free bytes but no inodes.
• KubePersistentVolumeErrors: an expansion or reprovision went wrong.
• KubePodCrashLooping: the typical downstream symptom once writes fail.