Docker build cache debugging: Fix slow incremental builds

We’ve all been there: you change a single line of code in your application, run docker build, and watch helplessly as the entire process takes six minutes because every layer re-runs. It’s a classic workflow killer. After spending way too much time staring at build logs during an on-call rotation last month, I realized our team was constantly breaking the docker build cache by ignoring the simple rules of layer ordering.

Why Your Cache Keeps Breaking

Docker builds images as a series of layers. When you run a command in your Dockerfile—like COPY or RUN—Docker checks if that exact command has been executed before. If it has, and the files it depends on haven't changed, it uses the cached layer.

The most common mistake is placing high-frequency change commands before low-frequency ones. If you COPY . . before installing your dependencies, every single file edit invalidates your npm install or pip install layer. That’s roughly 2-3 minutes of wasted time per build, just waiting for packages to download again.

The Right Way to Order Your Dockerfile

To fix this, you need to be intentional about your layer structure. We moved our dependency installation steps to the top, right after copying only the manifest files.

Dockerfile
# Bad approach: COPY . . first invalidates everything
FROM node:18-alpine
COPY . .
RUN npm install

# Good approach: Cache dependencies separately
FROM node:18-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm install
COPY . .

By copying only the package manifests, the RUN npm install layer only rebuilds when your dependencies change. Since your code changes far more often than your dependencies, this simple shift saves about 90 seconds on every incremental build. If you're looking for more ways to tighten up your images, check out my guide on Docker image optimization.

Debugging with BuildKit

If you're still seeing cache misses, it's time to dig into the buildkit cache metadata. BuildKit provides much better feedback than the legacy builder. You can enable it by setting DOCKER_BUILDKIT=1 in your environment, though it's the default in modern Docker versions.

Run your build with the --progress=plain flag to see exactly which layer is failing to hit the cache:

Bash
docker build --progress=plain -t my-app:latest .

Look for lines that say CACHED versus those that don't. If you see a layer re-running that shouldn't be, check the COPY commands immediately preceding it. Often, a hidden file—like a .git folder or a local log—is sneaking into the build context and triggering a change. Use a .dockerignore file to keep your build context clean.

Comparing Build Strategies

When optimizing for speed, you have a few levers you can pull. Here’s how they compare:

For production-grade pipelines, I highly recommend combining these with Docker multi-stage builds to ensure your final images are as small as possible.

What I’d Do Differently Next Time

I spent too much time manually pruning layers when I should have been using remote caching in our CI/CD pipeline from day one. If you're working in a team, local cache isn't enough; you need a shared cache that your GitHub Actions or GitLab runners can pull from. It’s a bit of a headache to configure initially, but it’s the only way to get sub-minute build times across a distributed team.

Also, don't forget to periodically audit your environment. I wrote a deep dive on Docker build cache: How to Audit and Clean for Faster CI/CD that covers how to clear out the junk that accumulates over time.

Are you still seeing cache misses? Check your Dockerfile for non-deterministic commands like RUN date or RUN apt-get update without a version pin. Those will break your dockerfile optimization efforts every single time.

FAQ

Q: Why does my RUN npm install still run even when nothing changed? A: It’s likely because your COPY . . command is earlier in the file. Ensure you are only copying the package.json file before the install command.

Q: Does BuildKit work on older Docker versions? A: It was introduced in Docker 18.09. If you're on an older version, you're missing out on significant performance gains—it's time to upgrade.

Q: How do I know if my cache is actually being used? A: Run your build twice. The second build should show CACHED for every single step. If it doesn't, you have a non-deterministic process or an invalidation issue.