During a routine security audit on our production EKS cluster last month, we discovered that our API server was leaking sensitive metadata to unauthorized service accounts. We weren't logging enough detail to reconstruct the timeline of the privilege escalation attempt, which took us roughly 42 hours of manual investigation across CloudWatch logs and VPC flow logs to fully map. It was a wake-up call. If you aren't actively monitoring your cluster's heartbeat, you're essentially flying blind.

Implementing Kubernetes Audit Logs for API Server Security

To gain visibility, we first focused on enabling Kubernetes Audit Logs. These logs are the source of truth for every request that hits your API server. Without them, threat detection is just a guessing game.

We defined a strict audit policy in a YAML file. Here is the configuration we landed on after a few failed attempts that crashed our logging sidecar due to excessive volume:

YAML
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["secrets", "configmaps"]
  - level: Metadata
    omitStages:
    - "RequestReceived"

We initially tried logging everything at RequestResponse level. That was a disaster; it increased our log ingestion costs by nearly 250% and triggered rate-limiting on our log aggregator. We scaled it back to Metadata for most resources, keeping RequestResponse only for sensitive objects like Secrets.

Integrating Falco for Real-time Threat Detection

Once the logs were flowing, we needed a way to act on them. This is where Falco becomes essential. While audit logs tell you what happened, Falco helps you identify the "why" and "who" in real-time. If you haven't already, you should look into Kubernetes Security: Detecting Anomalous Behavior with Falco and eBPF to understand how eBPF complements this log-based approach.

We deployed Falco via Helm 3.12.3. The key is to map your audit log sink to the Falco input stream. We used a local file-based sink on the master nodes, which Falco then consumes.

The configuration in falco.yaml looks like this:

YAML
auditLog:
  enabled: true
  hostname: "k8s-api-server"
  file: "/var/log/kubernetes/audit.log"

By connecting these two, we created a feedback loop. When the API server processes a kubectl exec request into a sensitive namespace, Falco fires an alert within 280ms. It’s significantly faster than waiting for a log aggregator to index the event.

Why This Matters for Kubernetes Security Monitoring

You can't rely on perimeter security alone. In a modern cluster, the API server is the primary attack surface. Just as I discuss in Kubernetes Security Auditing: Automating Trivy with Admission Controllers, you need multiple layers of defense. Audit logs provide the historical record, while Falco provides the immediate reaction.

We also experimented with piping these logs into a SIEM. We hit a wall with the parsing logic—the JSON structure of Kubernetes audit logs is notoriously complex. We eventually wrote a custom fluent-bit filter to normalize the fields before sending them to our backend, which saved us about 30% on storage costs by dropping unnecessary debug stages.

FAQ

Q: Does enabling verbose audit logs impact API server performance? A: Yes, it can. If you log every request at the RequestResponse level, you will see increased CPU usage on your control plane. Always start with Metadata and only escalate to RequestResponse for critical resources.

Q: Can Falco replace my SIEM for Kubernetes Security Monitoring? A: Not entirely. Falco is excellent for runtime threat detection and immediate alerting, but it lacks the long-term archival and complex correlation capabilities of a dedicated SIEM.

Q: What is the biggest mistake people make with Kubernetes Audit Logs? A: Over-logging. Most teams try to log everything and end up drowning in noise, which makes it impossible to spot actual security incidents when they happen.

I'm still not entirely convinced that our current retention policy—90 days—is sufficient for compliance requirements. We might need to move to a tiered storage approach next quarter. We're also looking at whether we can offload some of this detection to an eBPF-based tool to reduce the dependency on log file parsing, but that's a project for another sprint.