Fractal Techware

Alert runbooks /

BlackboxSslCertificateWillExpireSoon

An endpoint probed over TLS is serving a certificate chain that expires soon.

   
Severity warning, critical
Source blackbox_exporter /probe (http or tcp module with TLS)
Key metric probe_ssl_earliest_cert_expiry (Unix time of the first certificate in the chain to expire)

What it means

On every TLS probe the exporter records when the soonest-expiring certificate in the presented chain runs out. The alert compares that with the current time. The warning fires a couple of weeks ahead, when there is time for a normal renewal. The critical fires when only a few days remain and an outage is imminent.

Once the certificate expires, browsers and API clients refuse to connect, so this is a guaranteed outage with a known date.

Common causes

First checks

  1. List certificates by days left:
    sort((probe_ssl_earliest_cert_expiry - time()) / 86400)
    
  2. Inspect what the endpoint serves right now:
    openssl s_client -connect <host>:443 -servername <host> -showcerts </dev/null 2>/dev/null \
      | openssl x509 -noout -subject -issuer -enddate
    

    If the leaf looks fine, check the intermediates printed by -showcerts.

  3. Use the exporter debug output to see the chain it received:
    curl -s "http://<blackbox-exporter>:9115/probe?target=https://<host>&module=<module>&debug=true"
    
  4. For cert-manager managed certificates:
    kubectl get certificate -A
    cmctl status certificate <name> -n <namespace>
    

Fixing it

Fix the renewal path (challenge, DNS credentials, rate limits), then force renewal, for example cmctl renew <name> -n <namespace>. Make sure the ingress or server picks up the new secret; some need a reload. Replace an expiring intermediate by serving the chain your CA currently provides.