Linux Docker Inotify Auditing: Real-Time File Monitoring with Systemd

Last month, a rogue process inside one of my production containers started overwriting environment configuration files, causing a cascade of failures across our microservices. I needed a way to catch these writes the moment they happened, without installing heavy agents inside the image.

The most efficient way to achieve this is by tapping into the Linux kernel’s inotify subsystem directly from the host. By combining this with systemd, you can create a lightweight, zero-dependency auditing layer that monitors your Docker volumes in real-time.

Why Linux Inotify is your best bet for Docker auditing

When you mount a volume in Docker, the files exist on your host's filesystem. This means you don't need to pollute your container with monitoring tools. You can watch the host directory directly.

inotify (inode notify) is a kernel subsystem that notifies userspace applications of filesystem events. While tools like AIDE are great for periodic checks—which I’ve covered before in Linux Security: File Integrity Monitoring with AIDE and Systemd Timers—they aren't designed for the millisecond-level reaction times needed for active incident response.

We first tried using standard log tailing, but that failed because the application logs didn't capture file metadata changes or unauthorized permission shifts. Switching to inotifywait gave us the granularity we needed to see who changed the file and what process initiated the write.

Setting up real-time Docker Inotify monitoring

To get started, you’ll need inotify-tools installed on your host. On Debian or Ubuntu, it’s a simple sudo apt install inotify-tools.

The core of our monitor is a small shell script that watches a specific Docker volume path. Here’s a basic implementation:

Bash
#!/bin/bash
# monitor_docker_dir.sh
WATCH_DIR="/var/lib/docker/volumes/my_app_data/_data"

inotifywait -m -e modify -e create -e delete "$WATCH_DIR" | while read path action file; do
    echo "$(date): $action detected on $file" | logger -t docker-audit
done

This script runs in the background and pipes events to the system journal. You can then query these events easily using journalctl -t docker-audit.

Managing the monitor with Systemd

Running a script in a tmux session is fine for debugging, but for production, you want the robustness of systemd. This ensures your monitor starts on boot and restarts if it crashes.

Create a service file at /etc/systemd/system/docker-audit.service:

INI
[Unit]
Description=Real-time Docker Volume Auditor
After=docker.service

[Service]
ExecStart=/usr/local/bin/monitor_docker_dir.sh
Restart=always
User=root

[Install]
WantedBy=multi-user.target

After creating the file, run systemctl daemon-reload and systemctl enable --now docker-audit. Your auditing layer is now a first-class citizen on your Linux host.

Important trade-offs and limitations

This approach is powerful, but it’s not a silver bullet.

1. Kernel limits: If you are monitoring thousands of files, you might hit the fs.inotify.max_user_watches limit. You can check your current limit with cat /proc/sys/fs/inotify/max_user_watches. If you need more, bump it to 524288 in /etc/sysctl.conf.
2. Performance: inotify is extremely efficient, but spawning a process for every single event can cause CPU spikes if your application writes to the watched directory constantly. If you're logging high-frequency data, exclude those specific subdirectories from your watch list.
3. Container isolation: Remember that Rootless Docker: Secure Your Containers Without Root Privileges changes where files live on the host. If you’re running rootless, your mount points will be in your home directory, not under /var/lib/docker.

FAQ: Common questions about Docker auditing

Does this affect container performance? Not at all. The inotify subsystem runs at the kernel level. As long as you aren't logging to a slow disk, the overhead is negligible.

Can I use this to block writes? inotify is an observability tool, not a security enforcement tool. It tells you that a write happened, but it doesn't stop it. If you need to block writes, look into seccomp profiles or AppArmor for your containers.

How do I handle alerts? The example above uses logger, which sends events to the local journal. In a real production setup, you should pipe these logs into a centralized system like ELK or Grafana Loki to get Slack or PagerDuty alerts.

I’m still experimenting with how to correlate these filesystem events with specific Docker container IDs. Currently, I’m cross-referencing the PID from the audit logs with docker inspect to map the filesystem activity back to the container, but it’s a bit manual. If you’re looking for a more automated way to manage your host's overall stability, I’d suggest reviewing your Linux Performance Tuning: Managing Swap and OOM for Docker VPS settings, as unexpected disk activity can sometimes trigger OOM issues if your logs grow too fast.

Monitoring is rarely a "set it and forget it" task. This inotify setup saved me roughly three hours of manual log digging during the last incident, but I’m still keeping an eye on the memory consumption of the monitor itself. Start small, verify your event volume, and adjust your watches accordingly.