eBPF-based Network Traffic Inspection for Docker Containers

Last month, I spent about three days chasing a phantom 150ms latency spike between two Docker containers on a standalone production server. Standard tools like tcpdump were giving me too much noise, and I didn't want to install heavy monitoring agents that might skew the results. I needed a way to see exactly what was happening at the kernel level.

That’s when I turned to eBPF. While most people associate it with complex service meshes, you can use it to get granular visibility into your host's networking stack without a full orchestration layer.

Why eBPF for Docker Debugging?

When you’re debugging Linux networking in a containerized environment, traditional packet capture often misses the context of the container itself. You end up with a wall of IP addresses and no idea which container belongs to which socket.

eBPF changes the game by letting you hook into the kernel's tracepoints. It’s significantly faster than traditional libpcap methods because it processes data right where the packets live. I’ve previously explored eBPF-based socket monitoring: Tracking latency in Docker containers to show how you can isolate per-container metrics without the overhead of sidecar proxies.

Setting Up Hubble-cli Outside of Kubernetes

Usually, hubble-cli is synonymous with Cilium and Kubernetes. However, the core power of eBPF-based observability isn't restricted to K8s. If you’re running a modern kernel (5.4+ is highly recommended), you can leverage the underlying eBPF maps to inspect traffic on a plain Docker host.

First, ensure you have the Cilium/Hubble binary installed. You don't need the full agent if you're just looking for raw visibility into the bridge interfaces:

Bash
# Verify kernel version
uname -r
# Install Hubble CLI
curl -L https://github.com/cilium/hubble/releases/latest/download/hubble-linux-amd64.tar.gz | tar xz && sudo mv hubble /usr/local/bin/

Once installed, you’ll want to point it at the bridge interface created by Docker. Usually, this is docker0.

Debugging Container-to-Container Latency

The real value of eBPF in this context is the ability to correlate packets with process IDs. When I was troubleshooting that 150ms delay, I used a simple trace to watch the flow between my web container and my database container.

If you are struggling with silent drops or unexpected delays, it is often worth comparing your findings with techniques from Docker networking latency: Debugging with eBPF and tcpretrans.

Here is how I typically structure my inspection:

1. Identify the veth pair: Docker creates a veth interface for every container. Find yours using ip link.
2. Filter by flow: Use the Hubble CLI to monitor flows specifically associated with the container IP range.
3. Analyze the RTT: Look for gaps between the SYN and ACK packets.

If you find that the latency is happening outside the application code, you might be dealing with iptables congestion or connection tracking (conntrack) saturation. In more complex setups, you might consider Kubernetes Network Policies Debugging with Cilium Hubble if you eventually plan to migrate these containers to a managed cluster, as the tooling transitions perfectly.

The Trade-offs

I’ll be honest: debugging without Kubernetes means you lose the rich identity-based metadata. Hubble is built to understand "Service A" talking to "Service B" by looking at K8s labels. On a plain Docker host, you’re mostly seeing IPs and ports.

I initially tried to use tcpdump to solve this, but the context switching killed my performance and I missed a intermittent race condition. Moving to eBPF-based inspection felt like turning on the lights in a dark room.

FAQ

Does this work on older kernels? Not reliably. eBPF features required for deep observability were heavily improved in 5.x kernels. If you’re on 4.15 or older, expect pain.

Is this safe for production? eBPF is designed for production safety. If your program crashes, the kernel verifier prevents it from taking down the system. However, don't run complex bpf_probe_write operations unless you really know what you're doing.

Can I use this for non-Docker processes? Yes. Since it hooks into the Linux networking stack, it sees everything—even host-level processes and system services.

I’m still refining my workflow. Next time, I’d probably automate the veth-to-container mapping using a simple script to save myself the manual lookup time. Tools evolve quickly, so keep an eye on the upstream kernel mailing lists if you’re doing this at scale.