Understanding Image Layers: Docker Caching and File Systems

Previously in this course, we covered Building Your First Image, where we turned a Dockerfile into a functional container image. In this lesson, we are going to look "under the hood" to understand why Docker builds are so fast and how they manage storage using layers.

The Concept of Image Layers

When you build an image, Docker doesn't just save a single, massive file. Instead, it creates a series of read-only layers. Every instruction in your Dockerfile—like RUN, COPY, or ADD—creates a new layer that represents the difference between the previous state and the new state.

Think of it like a stack of transparent sheets. Each sheet has a bit of information drawn on it. When you look at the stack, you see the final image. Because these layers are read-only, they can be shared between different images. If you have two images that both start with ubuntu:22.04, they both point to the same underlying layers on your disk. This approach saves massive amounts of space and makes "pulling" new images much faster.

Viewing Image History

Because every command in your Dockerfile maps to a specific layer, you can inspect exactly what happened during the build process using the docker history command.

Let’s look at the history of the image we built in our previous lesson. Run the following command in your terminal:

Bash
docker history <your-image-name>

You will see an output that looks something like this:

The top line is the most recent change. Each line represents a layer. If you see <missing> in the IMAGE column, don't worry—that's normal; it just means the layer was built on a different machine or doesn't have a unique ID in your local cache.

Identifying Cached Layers

The real power of this architecture is caching. When you run a docker build command, Docker checks each layer to see if it already exists in your local cache.

If you change a line in your Dockerfile (or change a file in your source code), Docker invalidates that specific layer and all subsequent layers. This is why the order of instructions in your Dockerfile matters significantly for performance.

1. Step 1: Docker runs the instruction.
2. Step 2: It generates a checksum of the command and the files involved.
3. Step 3: It compares the checksum to existing local layers.
4. Step 4: If it matches, it uses the cached layer (marked as Using cache in the terminal). If not, it executes the command and builds a new layer.

Hands-on Exercise: Triggering Cache Invalidation

To see this in action, follow these steps:

1. Build your project: Run docker build -t my-app:v1 . and note the time it takes.
2. Modify the Dockerfile: Add a harmless instruction at the end, like RUN echo "hello".
3. Build again: Run docker build -t my-app:v2 ..
4. Observe: Notice that all previous steps say Using cache, and only your new RUN instruction takes time.
5. Break the cache: Move that RUN instruction to the top of your Dockerfile (right after FROM).
6. Build again: You will notice that every subsequent step must be re-run because the cache was invalidated from the very first layer.

Common Pitfalls

• Adding large files early: If you COPY . . at the very beginning of your Dockerfile, any minor change to a single source file will invalidate your entire build cache, forcing Docker to re-run apt-get install or other time-consuming setup steps. Always copy only the files you need for specific steps.
• Assuming 'latest' is always new: Because Docker uses caches, if your base image changes on the registry, your local build might still be using an "old" cached layer. Use --no-cache if you suspect your build is stale.
• Layer Bloat: Every RUN command creates a layer. If you use ten RUN commands to install packages, you create ten layers. You can clean this up by chaining commands with && (e.g., RUN apt-get update && apt-get install -y package).

FAQ

Does deleting a container remove its layers? No. Layers are part of the image. Deleting a container only removes the "writable" layer associated with that specific instance.

How do I see the actual size of my layers? While docker history shows size, tools like Dive provide a visual, layer-by-layer breakdown of what files were added or modified.

Can I share layers across different projects? Yes. If two different projects use the same base image (e.g., node:18), Docker stores those layers exactly once on your machine.

Recap

We’ve explored how Docker optimizes builds through a layered file system. We learned that docker history reveals the blueprint of our image, and that caching is highly sensitive to the order of operations in our Dockerfile. By keeping our instructions efficient, we minimize build times and storage usage.

Up next: Optimizing the Build Cache