Alert runbooks /

KubeCronJobMissedSchedule

A CronJob’s scheduled run time passed a long time ago without a new Job being started, and the CronJob is not suspended.

What it means

kube-state-metrics computes when each CronJob should next run. If that time is far in the past, the CronJob controller did not create the Job. Suspended CronJobs are excluded, so this is an unintended skip.

Scheduled work (backups, cleanups, billing exports, certificate rotations) is silently not happening. These gaps are easy to miss for days.

Common causes

• concurrencyPolicy: Forbid with the previous run still active or hung, so each new run is skipped.
• Too many missed start times: after a long outage the controller refuses to schedule and logs a “too many missed start times” error; this happens when there is no startingDeadlineSeconds.
• startingDeadlineSeconds too short: a brief controller delay pushes the run past its deadline, so it is skipped.
• kube-controller-manager down or unhealthy, so no CronJob runs anywhere.
• Job creation rejected by a ResourceQuota or admission webhook.

First checks

1. See the last run and active Jobs:

   kubectl -n <namespace> get cronjob <cronjob>
   kubectl -n <namespace> describe cronjob <cronjob> | sed -n '/Events:/,$p'
   

2. How long since it last actually ran:

   time() - kube_cronjob_status_last_schedule_time{cronjob="<cronjob>"}
   

3. Find Jobs it owns, including any still running:

   kubectl -n <namespace> get jobs -o json \
     | jq -r '.items[] | select(.metadata.ownerReferences[0].name=="<cronjob>") | "\(.metadata.name) active=\(.status.active // 0) start=\(.status.startTime)"'
   

4. Check the schedule, time zone and policies:

   kubectl -n <namespace> get cronjob <cronjob> \
     -o jsonpath='{.spec.schedule} tz={.spec.timeZone} policy={.spec.concurrencyPolicy} deadline={.spec.startingDeadlineSeconds}{"\n"}'
   

5. If several CronJobs are late at once, check kube-controller-manager health and logs for cronjob errors.

Fixing it

If a previous run is hung, investigate and delete it so scheduling resumes. Run the missed work by hand: kubectl -n <namespace> create job <cronjob>-manual --from=cronjob/<cronjob>. Set startingDeadlineSeconds to a sensible window (for example a few minutes to an hour) so the controller recovers after outages instead of giving up. If the job is intentionally paused, suspend it properly so the alert stays quiet:

kubectl -n <namespace> patch cronjob <cronjob> -p '{"spec":{"suspend":true}}'

Related alerts

• KubeJobNotCompleted: a long-running run that blocks the next one.
• KubeJobFailed: runs start but fail.
• KubeControllerManagerDown: no controller, no scheduled Jobs.