Kubernetes Network Policies Debugging with Cilium Hubble

During our Q3 infrastructure migration, we hit a wall. A critical service in our production cluster suddenly stopped talking to our PostgreSQL instance. The metrics showed a clean 5xx error rate, but the logs were suspiciously quiet. We spent roughly 4 hours digging through standard iptables logs before realizing the issue wasn't the application—it was an overly aggressive network policy we’d applied hours earlier.

If you've been working with Kubernetes network policies, you know the pain of "black box" connectivity. You apply a YAML manifest, and either it works, or the traffic vanishes into the ether. That’s where eBPF-powered tools change the game.

Debugging with Cilium Hubble and eBPF

We rely on Cilium Hubble for deep network observability. Unlike traditional tools that rely on sidecars or kernel-level packet capturing that can crush your CPU, Hubble hooks directly into the kernel via eBPF. This allows us to see exactly where a packet is dropped—whether it’s a policy rejection, a connection reset, or an invalid handshake.

When we hit this issue, I didn't reach for tcpdump. Instead, I fired up the Hubble CLI:

Bash
# Observe traffic for the specific namespace
hubble observe --namespace backend-prod --pod service-a --follow

The output was immediate. I saw a stream of "Policy Denied" events. Hubble didn't just tell me the traffic was blocked; it showed me the exact rule ID that triggered the drop.

Why our first attempt failed

We initially tried to "fix" it by widening the CIDR range in our CiliumNetworkPolicy. That was a mistake. It didn't solve the core issue; it just masked the problem while creating a massive security hole. We were essentially using a sledgehammer when we needed a scalpel.

After that failed, we realized we needed to understand Kubernetes networking at the identity level. We moved from IP-based rules to label-based selectors, which allowed us to permit traffic based on the application identity rather than the ephemeral pod IP.

Implementation Steps

To get this level of visibility, ensure you have Hubble enabled in your Cilium installation. If you’re running Cilium 1.14+, the commands are straightforward:

1. Enable Hubble Relay:

   Bash
   helm upgrade cilium cilium/cilium --namespace kube-system \
     --set hubble.relay.enabled=true \
     --set hubble.ui.enabled=true
   

2. Access the UI: Port-forwarding the UI makes visualizing these flows much easier than parsing terminal output:

   Bash
   cilium hubble ui
   

3. Verify Policy Drops: Keep an eye on the verdict field in the Hubble output. A DROPPED verdict with a Policy reason is your smoking gun.

The Reality of Network Observability

Using eBPF for Kubernetes security debugging is powerful, but it’s not magic. It generates a massive amount of data. In our cluster, we saw around 450 events per second during peak load. If you don't have a strategy for log aggregation—like Kubernetes logging—you’ll burn through your storage in hours.

We eventually narrowed the drop down to a missing toPorts definition in our policy. The service was trying to reach the DB on port 5432, but our policy only allowed traffic on port 80. It was a classic "oops" moment that standard Kubernetes tools would never have caught.

FAQ

Q: Does Hubble impact cluster performance? A: Because it uses eBPF, the overhead is minimal. We've seen roughly a 2-3% increase in CPU usage on nodes with high traffic, which is well worth the trade-off for the level of visibility provided.

Q: Can I use Hubble without Cilium? A: No, Hubble is deeply integrated with the Cilium data plane. You need Cilium as your CNI to get the eBPF hooks that Hubble requires.

Q: Is this better than standard NetworkPolicy logs? A: Yes. Standard Kubernetes logs are often inconsistent across different CNI providers. Hubble provides a consistent, high-fidelity view across the entire cluster.

Final Thoughts

I'm still not 100% satisfied with our current policy management. While Hubble helps us debug, we're still manually writing these manifests. Next time, I think we need to look into automating the policy generation process using tools that can "learn" from existing traffic patterns. It's easy to get lost in the weeds of labels and selectors, and I’m sure we’ll break connectivity again if we don’t move toward a more declarative, automated approach.