Docker Pruning Automation: Managing Orphaned Resources with Systemd

We’ve all been there: you run df -h on a production host only to see the disk usage at 98% because a rogue CI pipeline left behind thousands of dangling images. It's a classic case of "set it and forget it" biting back. While manual cleanup is fine for a dev machine, production infrastructure demands a more disciplined approach to Docker resource management.

If you’re already comfortable with Systemd Timers: The Better Way to Handle Linux Automation, you know they offer better observability and logging than traditional cron jobs. Let’s leverage that to build a robust cleanup routine that keeps your host healthy without constant manual intervention.

The Problem with "Simple" Pruning

Most engineers start by throwing docker system prune -f into a crontab. It works, until it doesn't. A broad prune can accidentally delete volumes or cache layers that a long-running background process actually needs. If you're managing GitHub Actions Self-Hosted Runners: Scaling Ephemeral Docker Containers, for instance, you don't want to nuke the build cache while a job is mid-flight.

We need a layered approach. Instead of a sledgehammer, we’ll use a combination of event-driven triggers and scheduled maintenance.

Implementing Docker Pruning with Systemd

To handle Linux Administration tasks like this properly, we’ll split the logic into two parts: the service that executes the command and the timer that triggers it.

Create /etc/systemd/system/docker-prune.service:

INI
[Unit]
Description=Cleanup orphaned Docker resources
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/bin/docker system prune -a --filter "until=24h" --force

I prefer using the --filter "until=24h" flag. It ensures that only resources older than a day are removed. It’s a small safety net that has saved me about two or three times from accidentally deleting an image I was currently debugging.

Next, create the timer at /etc/systemd/system/docker-prune.timer:

INI
[Unit]
Description=Run Docker prune weekly

[Timer]
OnCalendar=weekly
Persistent=true

[Install]
WantedBy=timers.target

Enable and start it:

Bash
systemctl daemon-reload
systemctl enable --now docker-prune.timer

Moving to Event-Driven Cleanup

Scheduled timers are great, but sometimes you need immediate feedback. If your server hosts ephemeral environments, you might want to trigger a cleanup immediately after a specific container exits.

This is where docker events shines. You can stream Docker events and pipe them into a script.

Bash
docker events --filter 'event=die' --format '{{.ID}}' | while read container_id; do
    # Trigger selective cleanup logic here
    docker rm -v $container_id
done

This is much more aggressive. I only recommend this if you have clear lifecycle management for your containers. If you're building out Docker-in-Docker CI Runners: Orchestrating Ephemeral Linux Environments, this approach is excellent because it ensures the host stays clean as soon as the job finishes.

Infrastructure Automation Best Practices

When you scale this to multiple nodes, manual systemd files become a liability. We’ve found that using Ansible to deploy these unit files across a fleet is the only way to keep configuration drift under control.

Here are the key takeaways for your Infrastructure Automation strategy:

1. Be Specific: Always use filters like --filter "until=24h" to avoid deleting active cache.
2. Use Timers: They integrate with journalctl, making it dead simple to check when the last cleanup happened.
3. Monitor: If your disk usage is still high, look into Linux performance: Managing Entropy Issues in Docker Containers or other resource bottlenecks that might be masking as "orphaned" storage issues.

FAQ: Common Gotchas

Q: Will this remove my persistent volumes? A: No. By default, docker system prune does not remove volumes unless you explicitly add the --volumes flag. Keep that flag off unless you are 100% sure you don't need the data.

Q: Can I use this on a single-node cluster? A: Absolutely. It’s actually more important on single-node setups because you don't have the luxury of offloading tasks to other nodes if one runs out of disk space.

Q: Is there a better way to check the status? A: Yes. Use systemctl list-timers to see when your next prune is scheduled. It’s much more readable than crontab -l.

I'm still tinkering with whether to move these tasks into a sidecar container instead of host-level systemd units. Sidecars are more portable, but they require the container to have the Docker socket mounted, which is a security trade-off I'm not always comfortable with. For now, native systemd units remain the most predictable path for my production hosts.