Bandwidth Throttling with eBPF and Linux Traffic Control

Last month, a single background job in one of our Docker containers decided to saturate the host's 1Gbps uplink. The resulting latency spikes knocked our primary API offline, costing us roughly 45 minutes of downtime before we identified the culprit. We’ve all been there: a runaway process or an aggressive backup task ruins the performance for every other service sharing the same host.

While Cgroup v2 is great for CPU and memory, managing network throughput at the container level often feels like a guessing game. We initially tried simple rate-limiting in our Nginx ingress, but that only handled HTTP traffic. It did nothing for our database syncs or background worker communication. We needed a deeper, host-level solution. That’s where eBPF and Linux Traffic Control (tc) come in.

Why use eBPF for Network Traffic Shaping?

If you've spent any time debugging, you might have already explored eBPF-based network traffic inspection for Docker containers to see what’s going on. However, seeing the traffic isn't the same as controlling it.

Standard tc commands are powerful, but they are notoriously difficult to map to specific Docker containers because container network interfaces (veth pairs) are dynamic. They disappear when the container stops and get recreated when it starts. By leveraging eBPF, we can attach programs directly to the host's network interfaces and filter packets based on the container's cgroup ID. This bypasses the need to constantly update interface names in your shaping scripts.

Step 1: Mapping Cgroups to Traffic

To get started, we need the container's cgroup ID. You can find this by inspecting the container's control group path. On a standard Ubuntu 22.04 host using Cgroup v2, it usually looks like this:

Bash
cat /sys/fs/cgroup/system.slice/docker-<container_id>.scope/cgroup.procs

Once you have the ID, you can write an eBPF program using libbpf that attaches to the tc hook. The program checks the skb->sk->sk_cgroup_id field. If it matches your "noisy" container, you can then mark the packet with a specific class ID.

Step 2: Applying Bandwidth Throttling

With the packets marked, we use Linux Traffic Control to enforce the limit. We define a Hierarchical Token Bucket (HTB) qdisc on the main host interface (e.g., eth0).

Bash
# Create the root qdisc
tc qdisc add dev eth0 root handle 1: htb default 10

# Create a class for limited traffic (e.g., 50Mbps)
tc class add dev eth0 parent 1: classid 1:1 htb rate 50mbit

# Match the class ID set by eBPF
tc filter add dev eth0 protocol ip parent 1: handle 1 bpf obj-path /usr/lib/bpf/throttle.o sec classifier

This setup effectively implements bandwidth throttling without needing to touch the internal networking stack of the container. If you’re also seeing weird packet drops, remember that Docker networking latency: Debugging with eBPF and tcpretrans is a vital companion to this process, as you don't want to mistake your own throttling for actual packet loss.

The Trade-offs

I’ll be honest: this isn't a "set it and forget it" solution.

1. Complexity: You’re now managing eBPF bytecode. If you update your kernel version, you might need to recompile your programs.
2. Visibility: Standard tools like ifconfig or ip link won't show you the eBPF filters. You have to use bpftool net show to verify what’s actually attached.
3. Overhead: While eBPF is incredibly efficient, adding complex logic to your packet path can introduce a few microseconds of latency. For 99% of web apps, it’s negligible, but for high-frequency trading or real-time UDP streams, it’s worth benchmarking.

Before jumping into eBPF, check if you can solve the problem using standard tc on the veth interface directly. If your containers are long-lived, mapping the veth interface name to the container ID is much easier. But if your environment is dynamic—like a CI/CD build agent pool—the eBPF approach is the only one that stays sane long-term.

FAQ

Does this work with Overlay networks? Yes, but you have to attach the eBPF program to the underlying physical interface (like eth0) and account for the encapsulation overhead.

Can I use this to limit egress and ingress? Egress is easy. Ingress is trickier because the packet has already hit your host interface before it reaches the container. You’ll need to use an ingress qdisc, which is slightly more restrictive in what eBPF helpers you can use.

Is this similar to Cilium? Cilium essentially does this at scale using advanced eBPF maps. If you’re already running a large cluster, don't write your own—use their existing bandwidth manager. I only recommend the manual route for standalone Docker hosts where installing a full CNI is overkill.

I’m still experimenting with how to automate the cgroup ID lookups during container startup. Currently, I’m using a simple Python sidecar that watches the Docker socket and updates the eBPF map. It’s not perfect, but it’s stopped the outages.