Kubernetes Security: Implementing Runtime Defense with Tetragon and eBPF

During a routine audit following a suspicious spike in kubectl exec activity across our production namespace, we realized our logs were essentially blind to what happened after an attacker gained shell access. While we had already mastered Kubernetes network policies debugging with Cilium Hubble to control traffic, we lacked the visibility to see if a process was modifying files or escalating privileges inside the container. We needed something that operated deeper than traditional syscall monitoring, so we turned to Tetragon.

Implementing Kubernetes security with Tetragon and eBPF

Tetragon is a powerful tool because it uses eBPF to hook directly into the kernel, allowing us to observe and enforce policies at the runtime level. Unlike older security tools that rely on user-space agents, Tetragon executes its logic within the kernel, making it nearly impossible for a compromised process to bypass or hide its tracks.

We initially tried to stick with standard logging via Kubernetes security: detecting anomalous behavior with Falco and eBPF because of its mature rule set. However, we found that Falco's user-space processing introduced roughly 280ms of latency during high-volume event spikes, which wasn't acceptable for our latency-sensitive API gateways. Tetragon’s ability to perform "in-kernel" enforcement meant we could kill a process the moment it violated a policy, rather than just alerting on it after the fact.

Getting started with runtime security

To deploy Tetragon, we used the official Helm chart. The configuration requires careful tuning, especially if you're running on older kernels. We pinned our deployment to version 0.10.0 to ensure compatibility with our EKS nodes running kernel 5.10.

YAML
# Example Tetragon policy to block sensitive file access
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "block-etc-shadow-access"
spec:
  kprobes:
  - call: "sys_openat"
    syscall: true
    args:
    - index: 1
      type: "string"
    selectors:
    - matchArgs:
      - index: 1
        operator: "Equal"
        values:
        - "/etc/shadow"
      matchActions:
      - action: Sigkill

When we first applied this, we broke our own CI/CD pipeline. It turned out that a legacy sidecar container was attempting to read /etc/shadow during startup for authentication checks we didn't know existed. We had to backtrack, adjust the policy to exclude that specific namespace, and spend roughly two days refactoring the sidecar to use a proper secret injection pattern instead of direct file access.

Why container observability matters

True container observability is more than just traces and logs; it's about understanding the state of your kernel. By using eBPF, Tetragon gives us a real-time view of every system call, file access, and network socket connection. This is the missing piece of the puzzle when you've already implemented Kubernetes networking: implementing zero-trust with Cilium and Hubble but still feel exposed to local exploits.

The trade-off here is complexity. Managing Tetragon policies requires a deep understanding of the underlying system calls your applications make. If you get it wrong, you’ll trigger false positives that effectively DDOS your own services. We've learned to deploy policies in Audit mode for at least 48 hours before switching to Enforcement mode to avoid production outages.

FAQ

Does Tetragon replace Falco? Not necessarily. While Tetragon excels at enforcement and low-latency performance, Falco has a massive community-driven library of detection rules. Many teams use both.

Will Tetragon impact my node performance? Because it’s built on eBPF, the overhead is minimal. However, you should always monitor the CPU usage of the Tetragon daemonset, especially if you have highly aggressive tracing policies.

What is the biggest challenge with Tetragon? Writing the TracingPolicy definitions correctly. It requires knowledge of kernel syscalls, which isn't standard knowledge for most DevOps engineers.

I’m still not 100% comfortable with our current policy maintenance process. We’re currently exploring ways to automate policy generation based on observed behavior, but I worry that "learning" policies will just codify the bad habits we’re trying to eliminate. We’ll see how the next sprint goes.