Troubleshooting Network Policies: A Docker Guide

Previously in this course, we explored container health monitoring to track service states and automate container restarts. While monitoring application health keeps your services running, silent failures often stem from misconfigured routing rules and dropped packets. In this lesson, we add network policy inspection and debugging tools to our multi-service project, focusing on how to inspect network policies, test firewall rules, and verify inter-container connectivity.

When multiple containers communicate across isolated bridges, unexpected packet drops can cripple your stack. Whether you are dealing with default bridge limitations, custom user-defined networks, or advanced host-level iptables rules, knowing how to systematically troubleshoot network policies prevents hours of blind guesswork.

Understanding Docker Network Architecture and Packet Flows

Docker relies on Linux bridge devices, iptables, and user-defined namespaces to isolate traffic. When two containers talk on a custom network, traffic flows through a virtual ethernet pair (veth) into a Linux bridge (such as br-xxxx). If traffic fails, it is usually because a bridge rule, an external firewall, or an internal container port binding blocks the path.

Before diving into troubleshooting commands, let's look at how traffic flows between containers on a custom network and where policies enforce restrictions:

Flow diagram: Container A: app → veth pair Linux Bridge; Linux Bridge → iptables / FORWARD chain Network Policy / Rules; Network Policy / Rules → Allowed Container B: database; Network Policy / Rules → Dropped Connection Timeout / Reset

To diagnose these boundaries effectively, we must inspect the active network configuration, test individual firewall paths, and confirm container-to-container reachability.

Inspecting Network Configurations and Driver Settings

The first step in troubleshooting any connectivity issue is inspecting the active Docker network structure. Docker creates distinct subnets and assigns internal IP addresses to each attached container.

Let's inspect our project's active network using the docker network inspect command:

Bash
docker network inspect app_network

Look at the JSON output for the Containers block. It maps container names to their assigned internal IPv4 addresses. For example:

JSON
"Containers": {
    "a1b2c3d4e5f6": {
        "Name": "web-app",
        "IPv4Address": "172.20.0.2/16"
    },
    "f6e5d4c3b2a1": {
        "Name": "database",
        "IPv4Address": "172.20.0.3/16"
    }
}

If your web application cannot reach the database, verify that both containers are attached to the exact same user-defined network. Containers on the default bridge network cannot resolve each other by name, which is a common source of connectivity confusion. For lower-level kernel auditing on Linux hosts, combining these inspections with standard diagnostics (similar to techniques covered in Linux Network Troubleshooting: Mastering ss, netstat, and tcpdump) gives you absolute visibility into socket states.

Testing Inter-Container Connectivity and Firewall Rules

Once you have verified network membership and IP assignments, you need to test actual packet delivery. Do not rely solely on ping, as many modern applications disable ICMP or run stripped-down base images where ping is unavailable. Instead, test the exact application port using tools like nc (netcat), curl, or bash built-ins.

Let's run an interactive check from inside our web-app container to verify connectivity to the database container on port 5432:

Bash
docker exec -it web-app sh -c "nc -zv database 5432"

If the connection times out or is refused, inspect the container's internal routing table and active ports. If you need to audit how Linux maps ports and processes at the host level, refer to Network Ports and Services: A Linux Developer's Guide for foundational socket auditing techniques.

When dealing with strict environments, administrators often apply host-level iptables rules that interact with Docker's chains. You can view Docker's custom filter rules on the host machine using:

Bash
sudo iptables -L DOCKER-USER -v -n

The DOCKER-USER chain is specifically reserved for user-defined firewall policies, allowing you to safely inject custom drop or accept rules without Docker overwriting them during container lifecycle events.

Worked Example: Diagnosing a Blocked Multi-Service Route

Imagine our running multi-service project features a Node.js web container trying to query a PostgreSQL database. The application logs report persistent ECONNREFUSED errors. Let's troubleshoot this step-by-step.

1. Verify container running status and network attachment:

   Bash
   docker ps --filter "network=app_network"
   

2. Inspect the bridge network to capture internal IPs:

   Bash
   docker network inspect app_network --format='{{range .Containers}}{{.Name}} - {{.IPv4Address}}{{"\n"}}{{end}}'
   

3. Test DNS resolution inside the web container:

   Bash
   docker exec -it web-app nslookup database
   

   If DNS fails, Docker's embedded DNS server (127.0.0.11) may be failing to resolve the container name on a custom bridge.

4. Test raw TCP port connectivity:

   Bash
   docker exec -it web-app python3 -c "import socket; s = socket.socket(); s.connect(('database', 5432))"
   

By methodically checking network membership, DNS resolution, and raw TCP socket creation, you isolate whether the failure is a routing misconfiguration, a dead service, or a strict firewall policy.

Hands-on Exercise

To practice troubleshooting network policies, complete the following steps in your local environment:

1. Create a user-defined bridge network named debug-net:

   Bash
   docker network create debug-net
   

2. Spin up two lightweight Alpine containers attached to debug-net, keeping them alive with sleep:

   Bash
   docker run -d --name net-test-1 --network debug-net alpine sleep 3600
   docker run -d --name net-test-2 --network debug-net alpine sleep 3600
   

3. Use docker exec to run nc or wget from net-test-1 targeting net-test-2.
4. Inspect the debug-net network JSON to confirm both container IP allocations.

Common Pitfalls

• Using the default bridge network: Relying on the default bridge network prevents automatic container name resolution. Always use user-defined bridge networks for multi-service apps.
• Ignoring host firewall overrides: Forgetting that UFW or firewalld rules on the host can drop packets destined for Docker bridge interfaces.
• Assuming ping failure means complete isolation: Concluding that a network is down just because ICMP ping is blocked, when the actual application port (e.g., HTTP 80) is fully open and functional.
• Misinterpreting port publishing (-p) vs internal networking: Trying to use host port bindings for inter-container communication instead of internal container ports on the private bridge network.

FAQ

How do I check which containers are attached to a specific Docker network?

Run docker network inspect <network_name> and examine the Containers JSON object in the output, or format the output using Go templates as shown earlier in this lesson.

Why can't my container resolve another container's name?

This usually happens when containers are attached to different networks, or when you are using the legacy default bridge network which does not support built-in automatic internal DNS resolution.

How do I block traffic between two specific containers on the same network?

Docker's standard bridge driver allows all inter-container communication by default. To enforce strict isolation, create separate user-defined networks or configure custom rules in the host's iptables DOCKER-USER chain.

Recap

Troubleshooting network policies requires a systematic approach: inspecting network attachments, verifying internal DNS and IP assignments, testing raw TCP sockets, and auditing host-level firewall rules. By mastering these diagnostic steps, you can quickly isolate and resolve inter-container communication failures in your multi-service deployments.

Up next: Image Signing and Notary