Debugging Docker Compose Healthcheck: Solving Startup Race Conditions

Last week, my team spent about three hours chasing a ghost in our staging environment. Our Node.js service kept crashing on boot because it couldn't connect to our PostgreSQL database, even though the database container technically reported as "up." It turns out, the database container was ready for connections, but it wasn't ready to accept queries yet.

We were hitting classic service orchestration race conditions. Docker Compose starts containers in parallel by default, and depends_on only tracks the container's start state, not its internal readiness. If you've been fighting similar connectivity issues, you're likely dealing with the gap between "container started" and "application ready."

Implementing a proper Docker Compose Healthcheck

The most reliable way to handle this is by moving beyond simple dependency flags. You need to implement an actual docker compose healthcheck that checks the internal state of your service before the dependent container starts.

Here is how I configure a robust check for a database service in docker-compose.yml:

YAML
services:
  db:
    image: postgres:15
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5
  
  app:
    build: .
    depends_on:
      db:
        condition: service_healthy

By setting condition: service_healthy, Docker Compose forces the app container to wait until the db health check returns a zero exit code. This is the single most effective way to eliminate startup race conditions in your local and CI environments.

Why service orchestration race conditions happen

It's tempting to think that depends_on is enough. I’ve been there. The reality is that the Docker daemon only cares if the entrypoint process is running. If your database container starts but takes 10 seconds to run its initialization scripts, your application container will try to connect during that window and fail.

Before we standardized on health checks, we used to rely on sleep commands in our entrypoint scripts. That's a bad pattern. It slows down your startup time significantly and often fails anyway if the database takes longer than your arbitrary sleep duration.

If you're still seeing connectivity errors, it's worth brushing up on Docker Compose Networking: How to Fix Service Connectivity Issues to ensure your bridge networks aren't causing secondary routing problems.

Advanced Docker health check debugging

When things go wrong, you need to see what the Docker engine sees. The first thing I do is inspect the container's state using the CLI:

Bash
docker inspect --format='{{json .State.Health}}' <container_id>

This command prints a JSON blob showing the last few health check attempts. If the status is unhealthy, the Log field will contain the stdout/stderr from your check command. It’s usually a permission issue or a missing binary inside the container.

Here is a quick comparison of how we handle startup dependencies:

Pro-tips for avoiding common pitfalls

Don't make the health check too aggressive. If your interval is set to 1s and your timeout is also 1s, you might trigger a false negative if the CPU spikes during container boot. I usually stick with an interval of 5s and a timeout of 3s.

Also, remember that if your application requires specific internal libraries to be initialized before it can handle traffic, you might need to combine this with Mastering Laravel app helper: A Guide to Resolving Dependencies if you're working in a PHP context, as application-level dependency injection can sometimes mask or exacerbate these network-level race conditions.

Frequently Asked Questions

Why does my depends_on: condition: service_healthy get ignored? Ensure you are using at least Docker Compose V2. Older versions of the Compose specification didn't support the condition key, causing them to silently ignore the configuration.

How do I debug a health check that keeps failing? Use docker exec -it <container_id> <check_command> to run the health check manually inside the container. If it fails there, you know the command is wrong or the environment isn't configured as expected.

Can I use health checks for things other than databases? Absolutely. You can use them for any service that has an HTTP endpoint. Use curl -f http://localhost:8080/health || exit 1 as your test command.

I'm still tinkering with how to handle "partial" readiness—where a service is up but hasn't fully loaded its cache—but for most scenarios, the native health check pattern is the way to go. If you find yourself constantly pruning dead containers during these tests, don't forget to use Docker Compose Prune: How to Clean Up Orphaned Containers to keep your environment clean.