Fractal Techware

Guides /

Rolling out Kyverno from Audit to Enforce safely

Policy and PolicyException tested with ghcr.io/kyverno/kyverno-cli:v1.19.1 (kyverno test: 3 of 3 pass). Manifests checked with kubeconform against the Kyverno schemas. The kubectl report queries are standard Kyverno and Kubernetes commands; run them against your own cluster.

Turning on a new admission policy in Deny mode on a live cluster is how a platform team finds out, at 17:55 on a Friday, that the ingress controller’s DaemonSet used :latest. The safe path takes a few weeks, and most of that time is waiting while reports fill up.

Phase 0  CI only        kyverno test / apply against manifests in Git
Phase 1  Audit          violations recorded in PolicyReports, nothing blocked
Phase 2  Audit + Warn   developers see warnings on every kubectl apply / CI deploy
Phase 3  Deny           violations rejected; exceptions handled explicitly

Phase 0: test in CI before anything reaches the cluster

Every policy gets a kyverno test suite with passing and failing resources. See the examples for disallowing :latest and requiring requests and limits. Also run kyverno apply against your real rendered manifests (Helm template or Kustomize output) to find violations before they are deployed:

helm template my-app ./chart | \
  kyverno apply policies/ --resource - --remove-color

This is the cheapest phase. Every violation found here never becomes a production incident.

Phase 1: Audit

Install the policies with validationActions: [Audit]. Two things happen:

  1. Admission: new and updated pods are evaluated and the result is recorded, but nothing is blocked.
  2. Background scan: existing resources are evaluated too (spec.evaluation.background.enabled defaults to true). You see violations from workloads deployed months ago, not just new ones.

Results land in namespaced PolicyReport objects (short name polr), one per resource:

# Overview: pass/fail counts per resource
kubectl get polr -A

# Every failure as resource, policy, message
kubectl get polr -A -o json | jq -r '
  .items[] | (.scope.kind + " " + .metadata.namespace + "/" + .scope.name) as $res
  | .results[]? | select(.result == "fail")
  | [$res, .policy, .message] | @tsv'

Stay in Audit for at least a full release cycle, and a week at minimum, so that batch jobs, CronJobs and rarely deployed services show up. Then work through the list:

Phase 2: Audit + Warn

spec:
  validationActions: [Audit, Warn]

With Warn, the API server returns the policy message as a warning on every create or update. kubectl apply prints it, and so do Helm, Argo CD and most CI tools. Nothing is blocked yet, but the people who own the manifests now see the problem in their own workflow, without anyone opening a ticket.

Deny and Warn cannot be combined. The API rejects that pair because a denial already returns the message. Audit works with either.

Phase 3: Deny

When the failure count in PolicyReports is zero, or every remaining failure is covered by an exception, switch:

spec:
  validationActions: [Deny]

Roll this out one policy at a time, starting with the ones that had the fewest findings. Keep the previous mode in Git so reverting is a one-line change.

Two things to know about enforcement:

PolicyExceptions: exempt one workload, not a namespace

Some workloads really need what a policy forbids: a node exporter, a CNI agent, a vendor image you cannot rebuild. Instead of weakening the policy for everyone, create a narrow PolicyException (saved as exception.yaml for the test below):

apiVersion: policies.kyverno.io/v1
kind: PolicyException
metadata:
  name: legacy-exporter-latest-tag
  namespace: policy-exceptions
spec:
  expiresAt: "2027-03-31T00:00:00Z"
  policyRefs:
    - name: disallow-latest-tag
      kind: ValidatingPolicy
  matchConditions:
    - name: only-legacy-exporter-in-monitoring
      expression: >-
        object.metadata.?namespace.orValue('') == 'monitoring' &&
        object.metadata.name.startsWith('legacy-exporter')

Key points:

helm upgrade --install kyverno kyverno/kyverno -n kyverno \
  --set features.policyExceptions.enabled=true \
  --set features.policyExceptions.namespace=policy-exceptions

Test exceptions exactly like policies. With the disallow-latest-tag policy from the earlier guide, this suite proves the exception covers the one pod it should, and nothing else:

# resources.yaml
apiVersion: v1
kind: Pod
metadata: {name: legacy-exporter, namespace: monitoring}
spec:
  containers:
    - {name: exporter, image: "vendor/exporter:latest"}
---
apiVersion: v1
kind: Pod
metadata: {name: legacy-exporter, namespace: team-a}
spec:
  containers:
    - {name: exporter, image: "vendor/exporter:latest"}
---
apiVersion: v1
kind: Pod
metadata: {name: other-app, namespace: monitoring}
spec:
  containers:
    - {name: app, image: "vendor/app:latest"}

# kyverno-test.yaml
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: latest-tag-with-exception
policies:
  - disallow-latest-tag.yaml
exceptions:
  - exception.yaml
resources:
  - resources.yaml
results:
  - isValidatingPolicy: true
    policy: disallow-latest-tag
    kind: Pod
    resources: [monitoring/legacy-exporter]
    result: skip
  - isValidatingPolicy: true
    policy: disallow-latest-tag
    kind: Pod
    resources: [team-a/legacy-exporter, monitoring/other-app]
    result: fail

docker run --rm -v "$PWD:/work" -w /work ghcr.io/kyverno/kyverno-cli:v1.19.1 test . --remove-color
# Test Summary: 3 tests passed and 0 tests failed

The same pod name in another namespace, and another pod in the same namespace, still fail. That is the property you want to lock in with a test.

Pitfalls

Next steps