Kubernetes Security: Detecting Anomalous Behavior with Falco and eBPF

Running a Kubernetes cluster without runtime visibility is like driving at night with your headlights off. You might be fine for a while, but you won't see the hazard until it's already hit you. I’ve spent enough late nights debugging compromised containers to know that static scanning isn't enough. You need to know what’s happening inside your pods while they're running.

That’s where Falco comes in. By leveraging eBPF (extended Berkeley Packet Filter), Falco gives us a deep, performant look into system calls without needing to instrument our application code.

Why eBPF is the Game Changer

In the past, security tools relied on kernel modules. They were brittle, prone to crashing the kernel, and a nightmare to manage across different Linux distributions. eBPF changed that. It allows us to run sandboxed programs inside the kernel safely. Falco uses this to intercept system calls (like execve, open, or connect) and compare them against a set of rules.

Since eBPF runs at the kernel level, it’s incredibly fast. You get the visibility you need without the massive performance tax that traditional agent-based security tools often impose.

Setting Up Falco with Helm

I prefer managing Falco via Helm. It’s cleaner and makes version control for your security policies much easier. If you're running a standard GKE or EKS cluster, you'll want to ensure your nodes are running a kernel that supports eBPF (usually 4.14+).

First, add the Falco repository:

Bash
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Now, install it. I recommend enabling the eBPF probe explicitly to ensure you aren't relying on the kernel module fallback:

Bash
helm install falco falcosecurity/falco \
  --set driver.kind=ebpf \
  --namespace falco --create-namespace

Writing Your First Detection Rule

Falco’s power lies in its YAML-based rules engine. Let's say you want to detect when someone executes a shell inside a container—a classic sign of a breach.

Create a custom rule file:

YAML
- rule: Shell in Container
  desc: Detects an interactive shell inside a container
  condition: >
    evt.type = execve and
    evt.dir = < and
    container.id != host and
    proc.name in (bash, sh, zsh)
  output: "Shell detected in container (user=%user.name container_id=%container.id image=%container.image.repository)"
  priority: WARNING

Apply this by mounting it as a ConfigMap or updating your values.yaml in Helm. When a user runs kubectl exec -it <pod> -- /bin/bash, Falco will trigger an alert immediately.

Monitoring Anomalous Behavior

Threat detection is only half the battle; you need to see the alerts. In production, I never rely on the standard output. You should configure Falco to stream logs to an external collector like Elasticsearch, Datadog, or even just a Slack webhook.

For a quick test, you can watch the logs directly:

Bash
kubectl logs -f -n falco -l app.kubernetes.io/name=falco

When you trigger the shell rule, you’ll see the JSON output. That JSON is your ticket to automated incident response. If you're using a tool like FalcoSidekick, you can automatically trigger a Lambda function or a Kubernetes Job to isolate the pod the moment a "Critical" rule is violated.

Hard Lessons Learned

I’ve seen engineers turn on every single default rule in Falco. Don't do that. You’ll be buried in noise within an hour.

1. Start with "Audit" mode: Deploy Falco and monitor the volume of alerts for 24 hours.
2. Filter the noise: If your CI/CD pipeline runs apt-get or npm install inside a container, write exceptions for those specific processes.
3. Focus on high-fidelity rules: Prioritize rules that detect privilege escalation, unexpected network connections to external IPs, and sensitive file access (like /etc/shadow or /var/run/secrets/kubernetes.io/serviceaccount/token).

Final Thoughts

Kubernetes security isn't a "set it and forget it" task. It’s an ongoing process of tuning and observing. eBPF provides the raw data, and Falco provides the brain. By combining these, you move from being reactive to being proactive.

If you're not monitoring your system calls, you’re flying blind. Start small, verify your rule sets, and build your defense in layers. Your future self—the one who isn't waking up at 3 AM to a breach notification—will thank you.