eBPF-based File Access Auditing: Track Config Changes with bpftrace

I spent three hours on a Tuesday night chasing a ghost. A configuration file on one of my production nodes kept reverting, and the standard logs told me absolutely nothing. I knew someone—or something—was touching the file, but inotify was failing to capture the process ID consistently under heavy load. That’s when I stopped relying on userspace tools and started looking at the kernel.

If you’re managing production Linux hosts, you’ve likely dealt with the "who touched this file?" mystery. While I previously explored File Integrity Monitoring: eBPF and Fanotify for Docker VPS, I wanted something faster and more ephemeral for quick debugging. That’s where bpftrace shines.

Why eBPF for File Auditing?

Standard auditing tools like auditd are powerful, but they’re also heavy. They can generate massive log volumes, and configuring them to ignore noisy events while catching the "needle in the haystack" is often a nightmare.

eBPF (Extended Berkeley Packet Filter) changes the game by allowing us to run sandboxed code directly in the Linux kernel. We can hook into system calls like openat or write and filter events before they ever hit userspace. This provides Linux security visibility with almost zero performance impact.

Getting Started with bpftrace

Before we dive into the code, ensure you have the necessary tooling installed. On a Debian-based system (like Ubuntu 22.04 or 24.04), it's straightforward:

Bash
sudo apt update
sudo apt install bpftrace

You’ll need a kernel that supports eBPF (anything 4.18+ is fine, though 5.x+ is preferred).

Monitoring Sensitive Config Changes

Let’s say I want to monitor changes to /etc/ssh/sshd_config. I don't want to log every single file access on the system; I only want to see who is opening this specific file for writing.

Here is a simple bpftrace script to accomplish this:

BPFTRACE
#!/usr/bin/bpftrace

tracepoint:syscalls:sys_enter_openat
{
    $filename = str(args->filename);
    if (strcontains($filename, "sshd_config")) {
        printf("Process %s (PID %d) is accessing file: %s\n", comm, pid, $filename);
    }
}

Save this as monitor_ssh.bt and run it with sudo bpftrace monitor_ssh.bt. When you edit the file in another terminal, you’ll see the output instantly.

The Trade-off: Complexity vs. Performance

I initially tried using kprobes to hook directly into the VFS layer. While that was more comprehensive, it was overkill for a simple audit. I switched to tracepoints because they are stable, well-documented, and much less likely to crash your kernel if you make a typo in your script.

Implementing eBPF for Docker Host Security

When you’re running containers, the host kernel sees everything. If you're concerned about Docker host security, you need to be able to distinguish between an event happening inside a container and an event happening on the host filesystem.

bpftrace lets us inspect the process context. If you want to filter out noise from your containers, you can check the process namespace:

BPFTRACE
tracepoint:syscalls:sys_enter_openat
/pid != 0/
{
    // You can add logic here to filter by cgroup or namespace
    printf("Accessing: %s by %s (PID: %d)\n", str(args->filename), comm, pid);
}

This is significantly more performant than Linux Docker Inotify Auditing: Real-Time File Monitoring with Systemd because it hooks into the syscall entry point directly. You aren't polling a file descriptor; you're letting the kernel notify you when the event occurs.

Real-world Auditing Strategy

When I’m in the middle of an incident response, I don't just want to know that a file was accessed; I want to know what was written. While bpftrace is great for monitoring access, if you need to capture the content of the changes, you might need to combine this with Linux kernel security: How to harden your Docker host with LKRG for a layered defense.

Here are a few tips for your production audits:

1. Keep it focused: Don't monitor everything. Use if statements in your BPF code to filter by path or PID.
2. Output to JSON: If you're piping this to a log aggregator, use printf to format your output as JSON strings.
3. Watch the overhead: Even though eBPF is efficient, running complex logic on every syscall will eventually show up in your CPU usage. Test your scripts on a staging environment first.

FAQ

Does this slow down my server? Not significantly. eBPF programs are JIT-compiled and run in the kernel. Unless you're doing heavy string manipulation on every single syscall in a high-traffic environment, you won't notice the impact.

Can I use this to block access? bpftrace is primarily for observability. While you can use it to influence return values, it's not a security firewall. If you need to block unauthorized access, look into seccomp profiles or AppArmor instead.

What happens if my script crashes? The BPF verifier ensures your code is safe before it runs. If it tries to access memory it shouldn't, the kernel simply won't load the program. It won't panic your system.

Final Thoughts

Using bpftrace for file integrity monitoring has saved me from hours of log-diving. It’s a surgical tool. You don't always need a full-blown SIEM or a heavy agent to understand what your host is doing. Sometimes, you just need a few lines of eBPF code to see the truth.

Next time, I'm planning to experiment with capturing the buffer contents of these writes to see exactly what configuration values are being injected. I’m still cautious about the performance hit of copying large buffers to userspace, but for security auditing, it might be worth the trade-off.