Fractal Techware

Alert runbooks /

KubePersistentVolumeClaimLost

A PersistentVolumeClaim is in the Lost phase: it was bound to a volume, and that PersistentVolume object is gone.

   
Severity critical
Source kube-state-metrics v2.x
Key metric kube_persistentvolumeclaim_status_phase (labels namespace, persistentvolumeclaim, phase)

What it means

The claim still records a volumeName, but Kubernetes cannot find a PV with that name. Pods already running may keep their mount until they restart; any new pod using the claim will fail to start.

It is critical because this often means data is missing from Kubernetes’ point of view. The backend disk may still exist, and the goal is to reconnect it before anyone recreates the claim and provisions an empty volume.

Common causes

First checks

  1. List lost claims:
    kubectl get pvc -A | grep Lost
    
  2. Note the volume the claim expects:
    kubectl -n <ns> get pvc <pvc> -o jsonpath='{.spec.volumeName}{"\n"}'
    kubectl get pv <volume-name>
    
  3. Check the audit log or events for who deleted it and when:
    kubectl get events -A --field-selector involvedObject.name=<volume-name>
    
  4. Find the backend volume in your storage system or cloud console, using the CSI volume handle from a backup of the PV manifest, GitOps history or Velero backup.
  5. Identify pods that still mount the claim and avoid restarting them:
    kubectl -n <ns> get pods -o wide
    

Fixing it

If the backend volume still exists, recreate the PV with the same name, the original csi.volumeHandle, persistentVolumeReclaimPolicy: Retain and a claimRef pointing at the claim’s namespace, name and UID. The claim should return to Bound. If the backend is gone, restore from backup into a new claim. Protect against repeats by using Retain for important data and restricting who may delete PVs.