Immutable Backups: Protecting Docker Volumes with Restic and S3

When a ransomware script hits your server, your live production database is the first thing it encrypts. If your automated backup script runs with standard write permissions, the attacker will just encrypt your backups too. I learned this the hard way during an incident where a misconfigured container exposed the host filesystem, and my backup snapshots were wiped within minutes.

To prevent this, you need to stop thinking about "backups" and start thinking about "immutable storage." We're going to use Restic combined with S3 Object Locking to ensure that once a backup is written, it cannot be modified or deleted by anyone—not even root—for a defined retention period.

The Strategy for Immutable Backups

The goal is to move the authority of "what can be deleted" away from the server performing the backup. When you configure an S3 bucket with Object Locking, the bucket itself enforces the retention policy. Restic, which is my go-to tool for Docker data persistence: Backing up volumes with Restic and Cron, handles the deduplication and encryption before the data ever leaves your host.

If you haven't hardened your host yet, start by looking into Rootless Docker: Secure Your Containers Without Root Privileges. Reducing the attack surface is the prerequisite for any data protection strategy.

Configuring the S3 Bucket

You must create your S3 bucket (or compatible storage like MinIO or Backblaze B2) with "Object Lock" enabled at creation time. You cannot enable this on an existing bucket later.

1. Create a new bucket.
2. Enable "Object Lock" in the S3 settings.
3. Once the bucket is created, go to the "Object Lock" tab and set a default retention period. I usually set this to 30 days for my dev environments and 90 days for production.

This retention period is the "hard" limit. Even if an attacker gains sudo access on your production server and runs restic forget --prune, the underlying S3 objects will remain protected until the retention clock runs out.

Setting Up Restic for Immutable Storage

Restic works perfectly with S3 Object Locking because it doesn't try to "delete" files in the traditional sense when you run a prune operation; it simply marks them as eligible for deletion. However, with Object Locking, the S3 API will reject those deletion requests.

First, install Restic on your host. I typically use the version provided by the package manager, but ensure it's at least version 0.12.0 or higher to support modern S3 backend features.

Bash
# Verify your version
restic version

Next, initialize your repository pointing to your locked bucket:

Bash
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
export RESTIC_PASSWORD=your_secure_password

restic -r s3:s3.amazonaws.com/your-locked-bucket init

Automating the Backup Workflow

I prefer using a simple shell script triggered by a systemd timer rather than a standard cron job. It gives you better logging and easier integration with Linux Docker Inotify Auditing: Real-Time File Monitoring with Systemd if you want to trigger alerts on unauthorized file access.

Here is a basic backup script structure:

Bash
#!/bin/bash
# /usr/local/bin/backup-docker.sh

# Stop or snapshot the database containers if necessary
docker stop my-db-container

# Run the backup
restic -r s3:s3.amazonaws.com/your-locked-bucket backup /var/lib/docker/volumes/

# Restart containers
docker start my-db-container

# Forget old snapshots (S3 will block the deletion of locked blobs)
restic -r s3:s3.amazonaws.com/your-locked-bucket forget --keep-last 7 --prune

The prune command will likely return errors in your logs saying "Access Denied" for objects currently under the lock. This is exactly what you want. It means your security layer is functioning. You’ll have to accept that your storage costs will grow slightly until the lock expires on those old, pruned objects, but that is the price of true immutability.

Important Trade-offs

I’ve found that using immutable storage isn't a silver bullet. Here is what I’m still figuring out:

• Storage Costs: Because you cannot delete data until the lock expires, you need to budget for extra storage. If you back up daily and set a 30-day lock, you'll effectively be paying for 30 full copies of your data plus the deltas.
• Recovery Complexity: If you actually get hit by ransomware, you can't just "clear the bucket" to start fresh. You have to wait for the locks to expire or have a secondary account with special IAM permissions to disable the lock (which I highly recommend keeping in a separate, MFA-protected root account).
• Performance: Sending large Docker volumes to S3 can saturate your uplink. Always use restic's built-in bandwidth limiting if you're running this on a small VPS.

For deeper insights into keeping your system clean, check out Linux Security: File Integrity Monitoring with AIDE and Systemd Timers. Combining file integrity monitoring with immutable backups creates a robust defense-in-depth posture.

Frequently Asked Questions

Can I delete backups manually if I need to? No. That is the point of the "Immutable" part of the setup. If you need to delete data for compliance (GDPR, etc.), you have to use a specific S3 "Compliance Mode" vs "Governance Mode." Governance mode allows users with specific IAM permissions to bypass the lock; Compliance mode is absolute.

Does Restic support S3 Object Locking natively? Restic doesn't need to "know" about the lock. It sends standard S3 DeleteObject requests. The S3 bucket policy and Object Lock configuration handle the rejection of those requests automatically.

What happens if my Restic repository gets corrupted? The immutability actually helps here. If an attacker tries to corrupt the repository files, they are essentially trying to "edit" an existing object. Since the object is locked, the edit will fail, and your repository will remain in its last known good state.

I'm still refining my retention policies. Right now, I'm leaning toward keeping 7 days of daily snapshots and 4 weekly ones, with the S3 bucket lock set to 30 days. It feels like a good balance between cost and having a safety net that spans a full month of potential silent data corruption.