Kubernetes Networking: Implementing Zero-Trust with Cilium and Hubble

The Zero-Trust Reality Check

If you're still relying on IP-based firewall rules in your Kubernetes cluster, you're fighting a losing battle. Pod IPs are ephemeral, and in a dynamic environment, they’re basically useless for security. I’ve spent countless hours debugging "allow-list" sprawl that breaks every time a deployment rolls out.

True security in Kubernetes requires a shift to identity-based networking. We need to move away from "who has this IP?" to "what service is this?" Enter Cilium. By leveraging eBPF, Cilium provides a high-performance, identity-aware networking layer that’s perfect for enforcing Zero-Trust security.

Why Cilium and Hubble?

Cilium replaces the standard Linux iptables with eBPF, allowing us to inspect and filter traffic at the kernel level. It doesn't care about IP addresses; it cares about labels. When you define a policy, Cilium maps it to a security identity, which persists regardless of where the pod moves.

But how do you know your policies are actually working? That’s where Hubble comes in. Hubble provides deep visibility into your network flows. It’s the difference between flying blind and having a high-definition map of every connection in your cluster.

Step 1: Installing Cilium and Enabling Hubble

First, make sure you have the Cilium CLI installed. I’m currently using version 1.14.x. If you haven't installed Cilium yet, use the Helm chart:

Bash
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.14.5 \
  --namespace kube-system \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

Once installed, verify the connection:

Bash
cilium status --wait
hubble status

Step 2: Defining Zero-Trust Network Policies

In a Zero-Trust architecture, the default posture is "deny all." If a packet isn't explicitly allowed, it gets dropped. Let’s create a policy that allows the frontend service to talk to the backend service, but nothing else.

Create a file named policy.yaml:

YAML
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "allow-frontend-to-backend"
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP

Apply this policy:

Bash
kubectl apply -f policy.yaml

Because we didn't specify a global "allow-all" rule, Cilium will automatically drop any traffic to the backend that doesn't originate from the frontend.

Step 3: Observing with Hubble

This is where the magic happens. You’ve applied the policy, but how do you verify it? Run the Hubble CLI to observe the drops in real-time:

Bash
hubble observe --pod backend --verdict DROPPED

You’ll see the logs showing the source, destination, and the specific policy that triggered the drop. It’s incredibly satisfying to see your security rules in action. If an unauthorized pod tries to hit your backend, Hubble logs it instantly, giving you the audit trail you need for compliance and troubleshooting.

Hard-Won Lessons

I’ve learned the hard way that you should never implement "deny-all" policies in a production environment without testing them in a staging namespace first.

1. Start with 'Default Deny' at the namespace level: It’s safer to break things in staging than to have a production outage.
2. Use Hubble UI for visualization: While the CLI is great for quick checks, the Hubble UI helps you spot traffic patterns that you might have missed.
3. Keep policies granular: Don't just allow all traffic between namespaces. Specify the exact ports and protocols.

Final Thoughts

Implementing Zero-Trust isn't a one-time configuration; it's a continuous process of observation and refinement. By using Cilium and Hubble, you’re not just securing your cluster; you’re gaining the observability required to move fast without breaking things.

The move to eBPF-based networking is inevitable. The sooner you get comfortable with identity-based policies, the more resilient your Kubernetes infrastructure will be. Start small, monitor with Hubble, and build your Zero-Trust walls layer by layer.