Building a CIS hardening operator

Our infrastructure team had a recurring problem: manually running kube-bench on clusters, triaging findings into tickets, and watching the same misconfigurations reappear after node replacements. Audit cycles were painful. I built an operator to close the loop automatically.

The goal wasn’t to automate everything — it was to handle the deterministic stuff without human intervention and flag the rest clearly.

How it works

The operator introduces a NodeHardeningPolicy CRD that maps CIS controls to a remediation mode: enforce, audit, or skip. A DaemonSet on each node runs the actual checks through a privileged container that can read /proc, kubelet config, and systemd units. Findings feed back into node annotations; the controller reconciles from there.

Remediations split into two buckets. Immediate ones — sysctl values, file permissions, audit log flags — apply on the next reconcile loop. Deferred ones — anything that touches kubelet config and requires a node restart — queue up and run during a maintenance window, gated by an annotation on the node. The distinction is manual but explicit: a control either lands in ImmediateRemediation or DeferredRemediation in the policy spec.

What went wrong

The first version reconciled on node events. On any cluster with regular churn — autoscaler cycling nodes, rolling updates — this meant the controller was running full benchmark passes constantly. I switched to a scheduled reconcile, once an hour, jittered per node so the API server doesn’t get hit all at once. Obvious fix, should have started there.

The bigger issue was EKS. Managed node groups rewrite kubelet config on node replacement, so any changes the operator made to /etc/kubernetes/kubelet.conf would disappear after the next node rotation. The operator now detects the eks.amazonaws.com/nodegroup label and skips kubelet-level remediations entirely on those nodes, flagging them as ManualOnly in the policy status. You still see the findings — you just don’t get the false assurance that they’re fixed.

Where it lands

L1 CIS controls for Kubernetes 1.28 are covered. L2 is partial — etcd encryption and API server audit policy live at the cluster provisioning layer, not the node level, so they’re out of scope for this operator. Those live in the Terraform modules.

We pass audit now without the manual scramble before each cycle. The operator has been running in production for about six months; the main operational cost is keeping the control mappings updated as benchmark versions change.

If I were starting over: audit mode first, run it for a few weeks, understand which remediations cause noise in your environment before enabling enforcement. Some controls sound safe on paper and aren’t — particularly anything touching file ownership in directories that other tooling also writes to.