Docker Security: Implementing Immutable Infrastructure via Read-Only Root

Last year, I spent three days recovering a compromised VPS because an attacker managed to drop a persistent backdoor into /usr/local/bin. It wasn't a sophisticated exploit, just a standard script that survived a reboot because the host filesystem was wide open. After that, I moved toward immutable infrastructure.

If you’re running Docker in production, your host OS should ideally be a static platform for your containers. By mounting your root filesystem as read-only, you make it significantly harder for attackers to gain persistence, even if a container breakout occurs.

Why Immutable Infrastructure Matters for VPS Hardening

Most Linux distributions are designed to be mutable. You install packages, modify config files, and let daemons write state to /var. However, Docker hosts are unique; the "state" should ideally live in Docker volumes or external databases, not on the host's root partition.

When you implement a read-only root, you force the system to behave predictably. Any attempt to modify system binaries or inject malicious scripts into /etc will fail instantly. It’s the ultimate form of Docker security.

The First Attempt: The "Read-Only" Trap

I initially tried simply mounting the root partition with the ro flag in /etc/fstab. That was a disaster. The system wouldn't even boot because systemd and various logging daemons expect to write to /run, /var/log, and /tmp.

You can't just flip a switch. You have to build a "sandbox" for the OS components that must write data.

Implementing a Read-Only Filesystem

To make your root filesystem effectively read-only, you need to use tmpfs (a filesystem stored in RAM) for the directories that require write access. Here is the strategy I use on Debian and Ubuntu hosts:

1. Mount /var/log and /var/tmp as tmpfs.
2. Mount /run as tmpfs (this is usually default).
3. Use overlayfs for system configurations if you need to persist specific changes across reboots, though I prefer keeping the root image pristine.

Step-by-Step Configuration

First, identify the directories that need to be writable. In a standard setup, these are usually /var/log, /var/tmp, /etc/machine-id, and /var/lib/docker (if you aren't using a separate partition for Docker).

Add these lines to your /etc/fstab:

Bash
tmpfs   /var/log    tmpfs   defaults,noatime,mode=0755  0   0
tmpfs   /var/tmp    tmpfs   defaults,noatime,mode=1777  0   0

After updating fstab, you need to ensure the services don't crash when they find these directories empty. You'll likely need to create systemd tmpfiles configurations in /etc/tmpfiles.d/ to recreate the directory structure on every boot.

Challenges with Docker and Persistence

If you’re hardening your host, you might also be interested in Rootless Docker: Secure Your Containers Without Root Privileges to further limit the blast radius. But remember: if your host is read-only, Docker’s default storage location (/var/lib/docker) will fail.

You must mount a dedicated, writable partition (like an XFS or EXT4 volume) to /var/lib/docker. Never store your container images or volumes on the read-only root itself. This separation is a core tenet of immutable infrastructure and ensures your host OS remains clean while your container data persists safely on a separate disk.

Beyond the Filesystem: Hardening Layers

A read-only root is just one piece of the puzzle. I typically pair this with Linux kernel security: How to harden your Docker host with LKRG to catch runtime exploits that a read-only filesystem might miss.

If you are managing multiple nodes, ensure your Linux system administration practices remain consistent. I’ve found that using a local repository, as detailed in Linux administration: Building a Private APT/YUM Mirror for Docker, keeps the system predictable. If you can't install arbitrary packages from the internet, you've already won half the battle.

FAQ: Common Hurdles

Q: Does this break system updates? Yes, it does. You have to remount the root as read-write (mount -o remount,rw /) before running apt upgrade. It’s an extra step, but that's the point—it forces you to be intentional about system changes.

Q: What about /etc/? You can't mount /etc as tmpfs because the system needs those files at boot. If you need to protect /etc specifically, consider using chattr +i on sensitive files, though it’s less robust than a full read-only mount.

Q: Is it worth the complexity? For a public-facing VPS, absolutely. It adds about two hours of setup time but provides peace of mind that a compromised container can't easily turn your host into a permanent botnet node.

I’m still experimenting with using dm-verity to ensure the integrity of the root filesystem at the block level. It’s significantly more complex than fstab tweaks, but it’s the next logical step for true immutability. If you’re just starting, stick to tmpfs for the volatile directories first. It’s enough to stop 90% of the automated script-kiddie attacks I see in my logs.