Master Linux Docker Inotify auditing to track file changes in real-time. Learn how to bridge container events with Systemd for robust host-level security.
Last month, a rogue process inside one of my production containers started overwriting environment configuration files, causing a cascade of failures across our microservices. I needed a way to catch these writes the moment they happened, without installing heavy agents inside the image.
The most efficient way to achieve this is by tapping into the Linux kernel’s inotify subsystem directly from the host. By combining this with systemd, you can create a lightweight, zero-dependency auditing layer that monitors your Docker volumes in real-time.
When you mount a volume in Docker, the files exist on your host's filesystem. This means you don't need to pollute your container with monitoring tools. You can watch the host directory directly.
inotify (inode notify) is a kernel subsystem that notifies userspace applications of filesystem events. While tools like AIDE are great for periodic checks—which I’ve covered before in Linux Security: File Integrity Monitoring with AIDE and Systemd Timers—they aren't designed for the millisecond-level reaction times needed for active incident response.
We first tried using standard log tailing, but that failed because the application logs didn't capture file metadata changes or unauthorized permission shifts. Switching to inotifywait gave us the granularity we needed to see who changed the file and what process initiated the write.
To get started, you’ll need inotify-tools installed on your host. On Debian or Ubuntu, it’s a simple sudo apt install inotify-tools.
The core of our monitor is a small shell script that watches a specific Docker volume path. Here’s a basic implementation:
Bash#!/bin/bash # monitor_docker_dir.sh WATCH_DIR="/var/lib/docker/volumes/my_app_data/_data" inotifywait -m -e modify -e create -e delete "$WATCH_DIR" | while read path action file; do echo "$(date): $action detected on $file" | logger -t docker-audit done
This script runs in the background and pipes events to the system journal. You can then query these events easily using journalctl -t docker-audit.
Running a script in a tmux session is fine for debugging, but for production, you want the robustness of systemd. This ensures your monitor starts on boot and restarts if it crashes.
Create a service file at /etc/systemd/system/docker-audit.service:
INI[Unit] Description=Real-time Docker Volume Auditor After=docker.service [Service] ExecStart=/usr/local/bin/monitor_docker_dir.sh Restart=always User=root [Install] WantedBy=multi-user.target
After creating the file, run systemctl daemon-reload and systemctl enable --now docker-audit. Your auditing layer is now a first-class citizen on your Linux host.
This approach is powerful, but it’s not a silver bullet.
fs.inotify.max_user_watches limit. You can check your current limit with cat /proc/sys/fs/inotify/max_user_watches. If you need more, bump it to 524288 in /etc/sysctl.conf.inotify is extremely efficient, but spawning a process for every single event can cause CPU spikes if your application writes to the watched directory constantly. If you're logging high-frequency data, exclude those specific subdirectories from your watch list./var/lib/docker.Does this affect container performance?
Not at all. The inotify subsystem runs at the kernel level. As long as you aren't logging to a slow disk, the overhead is negligible.
Can I use this to block writes?
inotify is an observability tool, not a security enforcement tool. It tells you that a write happened, but it doesn't stop it. If you need to block writes, look into seccomp profiles or AppArmor for your containers.
How do I handle alerts?
The example above uses logger, which sends events to the local journal. In a real production setup, you should pipe these logs into a centralized system like ELK or Grafana Loki to get Slack or PagerDuty alerts.
I’m still experimenting with how to correlate these filesystem events with specific Docker container IDs. Currently, I’m cross-referencing the PID from the audit logs with docker inspect to map the filesystem activity back to the container, but it’s a bit manual. If you’re looking for a more automated way to manage your host's overall stability, I’d suggest reviewing your Linux Performance Tuning: Managing Swap and OOM for Docker VPS settings, as unexpected disk activity can sometimes trigger OOM issues if your logs grow too fast.
Monitoring is rarely a "set it and forget it" task. This inotify setup saved me roughly three hours of manual log digging during the last incident, but I’m still keeping an eye on the memory consumption of the monitor itself. Start small, verify your event volume, and adjust your watches accordingly.
Linux kernel security starts at the host level. Learn how to implement LKRG for Docker host hardening and detect runtime integrity threats effectively.
Read moreLinux networking forensics for Docker is simpler than you think. Learn how to use tcpdump and Wireshark to capture container traffic directly from the host.