ZFS Docker Storage: Cutting Disk Footprint with LZ4 Compression

Running a fleet of Docker containers on standard ext4 or xfs filesystems eventually leads to the same problem: massive, bloated /var/lib/docker directories. If you're managing a small-to-medium Linux infrastructure, you know the pain of watching disk usage climb as you pull layers and build images.

After hitting a wall with an aging server last year, I moved my Docker storage to ZFS. By leveraging transparent LZ4 compression, I managed to reclaim roughly 35% of my disk space while actually seeing an improvement in read-heavy workloads. Here is how you can do the same.

Why ZFS for Docker Storage?

Most Linux hosts default to overlay2 on ext4. It’s reliable, but it’s essentially "dumb" storage. ZFS changes the game by providing block-level compression, snapshots, and checksumming at the filesystem level.

When you use the zfs storage driver for Docker, every layer and volume is a dataset. Because LZ4 is incredibly fast and operates inline, the CPU overhead is negligible for modern servers. In my experience, the compression ratio is often better than what you’d get with gzip, and it doesn't throttle I/O throughput.

Preparing the Host

Before you dive in, ensure your kernel supports ZFS. If you're running Ubuntu or Debian, you'll need the zfsutils-linux package.

Bash
sudo apt update
sudo apt install zfsutils-linux

Note: If you're currently hardening your host, remember that Linux kernel security: How to harden your Docker host with LKRG should happen before you reconfigure your primary storage paths.

Implementing the Setup

To get ZFS working with Docker, you shouldn't just point it at a sub-directory. You need a dedicated ZFS pool.

1. Create the pool: Assuming you have a spare partition or disk (e.g., /dev/sdb1):

   Bash
   sudo zpool create -f -o ashift=12 -m /var/lib/docker/zfs dockerpool /dev/sdb1
   

   The ashift=12 is critical for modern SSDs to prevent performance degradation due to sector size alignment issues.

2. Configure Compression: Enable LZ4 compression on the pool. It’s transparent, meaning your applications won't even know it's there.

   Bash
   sudo zfs set compression=lz4 dockerpool
   

3. Configure Docker: You need to tell Docker to use the ZFS driver. Edit /etc/docker/daemon.json:

   JSON
   {
     "storage-driver": "zfs"
   }
   

4. Restart the Service:

   Bash
   sudo systemctl restart docker
   

Managing the Trade-offs

This isn't a silver bullet. While disk optimization is the primary goal, ZFS is memory-hungry. By default, the ZFS Adaptive Replacement Cache (ARC) can consume up to 50% of your system's RAM. If you're running this on a small VPS with 2GB of RAM, you will trigger an OOM killer event within minutes.

To prevent this, cap the ARC size in /etc/modprobe.d/zfs.conf:

TEXT
options zfs zfs_arc_max=536870912

This limits the cache to 512MB. You’ll have to balance this based on your available headroom.

Also, be aware that when things go sideways, debugging I/O can be more complex than standard filesystems. If you find your containers hanging, don't immediately blame the compression. Use tools like iostat or eBPF to monitor latency, as discussed in Linux performance: Resolving Docker I/O Bottlenecks with eBPF. Sometimes the bottleneck isn't the disk, but a misconfigured ZFS record size.

What I’d Do Differently Next Time

I initially tried to migrate an existing /var/lib/docker directory by simply copying files over. That was a mistake. ZFS datasets don't play well with pre-existing overlay2 structures. I ended up wiping the directory and starting fresh, which meant re-pulling all my images.

If you're doing this in production, plan for a maintenance window where you can rebuild your containers from scratch. It’s faster and cleaner than trying to "fix" an existing storage driver migration.

I’m still experimenting with ZFS record sizes. For databases, smaller record sizes (like 8k or 16k) often perform better, but for general-purpose Docker layers, the default 128k is usually fine. Just keep an eye on your compression ratios using zfs get compressratio dockerpool—if it’s sitting at 1.0x, you’re likely storing encrypted or already-compressed data, and you might want to rethink your configuration.