Linux Traffic Control: Rate-Limiting Docker Egress Traffic

Last month, a single backup container on one of our production hosts decided to saturate our 1Gbps uplink, pushing latency for our primary API up by roughly 280ms. It was a classic "noisy neighbor" scenario that reminded me why trusting container isolation for network resources is a mistake. If you're tired of rogue containers stealing your bandwidth, it's time to get comfortable with tc.

While tools like Bandwidth Throttling with eBPF and Linux Traffic Control offer modern, high-performance alternatives, sometimes you just need to reach for the classic primitives built into the kernel. Here’s how we use Linux traffic control to clamp down on outbound traffic at the container level.

Understanding the Docker Networking Challenge

When you launch a container, Docker creates a virtual Ethernet pair (veth). One end sits inside the container namespace, and the other is attached to a bridge (usually docker0) on the host. Because these interfaces are just standard Linux network devices, they are fully compatible with tc.

The problem is that these interfaces are dynamic. When a container restarts, its veth ID changes. You can’t just hardcode an interface name in a startup script and expect it to stick. We initially tried applying tc rules to the docker0 bridge itself, but that limits the total traffic of the entire bridge, not individual containers. That's fine for some setups, but it's too blunt for most of our microservices.

Practical Egress Shaping with tc

To apply egress shaping to a specific container, you need to identify the host-side veth interface associated with that container's PID.

1. Identify the Interface

First, find the container's PID:

Bash
docker inspect --format '{{.State.Pid}}' <container_id>

Next, use nsenter to peek into the container's network namespace or simply map the interface index. A simpler way is to look at the link index:

Bash
# Inside the container
cat /sys/class/net/eth0/iflink

This number corresponds to the index of the host-side veth pair. You can find the name on the host by matching that index:

Bash
ip link | grep <index>

2. Applying the Throttle

Once you have the interface name (e.g., veth12345), you can apply a Token Bucket Filter (TBF) to limit its egress bandwidth.

Bash
# Limit egress to 10mbit with a 32kb burst
tc qdisc add dev veth12345 root tbf rate 10mbit burst 32kbit latency 400ms

This command forces the kernel to queue packets, effectively creating a "speed limit" for that specific container. If you need to modify an existing limit, use replace instead of add.

Why tc Beats Higher-Level Limits

We’ve experimented with various user-space proxies, but they add overhead and latency. Using tc keeps the shaping in the kernel, which is exactly where it belongs. It’s significantly more efficient than routing traffic through a sidecar proxy just to count bytes.

However, it's not a silver bullet. If you're doing complex traffic inspection, you might find eBPF-based Network Traffic Inspection for Docker Containers more useful for debugging. Also, remember that tc is egress-only by default. If you need to limit ingress (download speeds), you’ll need to use an Intermediate Functional Block (IFB) device to redirect traffic, which adds a layer of complexity that often isn't worth the trouble unless you're building a multi-tenant platform.

Common Pitfalls

• Persistence: tc rules are volatile. If the container restarts, the interface is destroyed and recreated. You need to hook your tc configuration into your deployment pipeline or a post-start script.
• The "Burst" Parameter: If your burst size is too small, you'll see significant packet loss for bursty traffic like TLS handshakes. Don't set it too low; 32kbit or 64kbit is usually a safe starting point for standard web traffic.
• Interface Renaming: Don't rely on the interface name staying the same across reboots. Always resolve the veth index dynamically at runtime.

If you find yourself managing complex network policies, consider if your infrastructure has outgrown simple scripts. For most, this manual approach is sufficient, but if you're dealing with massive scaling, you'll eventually want to look into CNI plugins that handle bandwidth management natively.

I’m still not entirely convinced that managing this at the host level is the "cleanest" way to handle multi-tenancy, but it's the most reliable way to stop a rogue process from killing your host's network connectivity without needing a total architecture overhaul. Next time, I might try automating this with a small Go binary that watches for Docker events—but for now, a few lines of shell script keeps our API snappy.

Frequently Asked Questions

Q: Does tc work on Docker Desktop for Mac/Windows? No. Docker Desktop runs inside a hidden Linux virtual machine. You don't have direct access to the host's tc stack in the same way you do on a bare-metal Linux server or a standard VPS.

Q: Can I use tc to limit CPU usage? No. tc is strictly for network traffic. For CPU and memory, you should use Docker's built-in resource constraints (--cpus, --memory) or manage cgroups directly, similar to how you’d handle Docker I/O throttling: Control container performance with Cgroup v2.

Q: How do I verify the limit is working? Use tc -s qdisc show dev veth12345. It will show you the number of sent packets and, crucially, the number of dropped packets if the container is hitting the limit.