Kubernetes Backup Strategies: Implementing Velero and MinIO

During a routine cluster upgrade last month, a misconfigured Helm chart wiped out our primary database PVCs across three namespaces. We thought we had a safety net, but our existing snapshot-only strategy failed because the cloud provider’s CSI driver didn’t support cross-zone restoration for the specific volume type we were using. It took us roughly 4 hours of frantic manual recovery to get services back to a stable state, which was 3 hours and 45 minutes longer than our RTO. That incident forced us to move beyond simple volume snapshots and build a proper Kubernetes backup pipeline that we actually control.

Implementing Kubernetes Backup with Velero and MinIO

We needed a solution that was vendor-agnostic and didn't rely on proprietary cloud snapshots. We chose Velero paired with MinIO to act as our S3-compatible object storage backend. This setup gives us total control over our backup lifecycle, allowing us to store data on-premises or across different cloud providers without locking ourselves into a single ecosystem.

If you’re just starting, you should first review the foundational concepts of Kubernetes disaster recovery: Velero and Restic implementation guide to ensure you understand how Restic handles persistent volume backups when native snapshots aren't available.

The Setup Process

First, deploy MinIO within your cluster or on dedicated infrastructure. We used the official MinIO Helm chart. Ensure you create a dedicated bucket for Velero and generate the access credentials.

YAML
# minio-values.yaml
accessKey: "velero-user"
secretKey: "super-secret-key"
buckets:
  - name: velero-backups
    policy: none

Once MinIO is reachable, install the Velero CLI and initialize it in your cluster. We use Velero v1.12.0. The key here is the backupStorageLocation configuration, which points to your MinIO service endpoint.

Dealing with Failure

Our first attempt at this implementation was a disaster. We tried to use the default Velero S3 plugin with an incorrect endpoint configuration, which led to 403 Forbidden errors during every backup attempt. We spent an entire afternoon debugging S3 signature versioning until we realized the MinIO instance wasn't serving traffic on the expected port because of a missing Kubernetes Service selector.

Don't skip the step of verifying your bucket connectivity using mc (the MinIO Client) before you even touch the Velero installation. Once we fixed the networking, the backups started flowing smoothly. We’ve since standardized our approach to Kubernetes disaster recovery: Velero and Restic implementation guide to handle those edge cases where simple snapshots aren't enough.

Why S3-Compatible Storage Matters

Using S3-compatible storage like MinIO is a game-changer for disaster recovery. It allows us to treat our backup metadata and volume data as distinct objects that are easily portable. If the entire cluster goes down, we can spin up a new K8s environment, point Velero at the same MinIO bucket, and trigger a restore.

When you're scaling, keep an eye on your storage costs and performance. We found that backing up 500GB of persistent data took around 28 minutes over our internal network. If you find your cluster growing, you might also want to look into Kubernetes observability: implementing distributed tracing with Tempo to ensure your backup jobs aren't silently impacting your application performance.

FAQ

Is MinIO production-ready for Kubernetes backup? Yes, if you configure it with proper distributed storage backends (like local NVMe drives or networked block storage) and ensure you have high availability enabled.

Can I use Velero without S3-compatible storage? Velero requires an object store to house the backup metadata and volume snapshots. While you can use cloud-native providers like AWS S3 or GCS, MinIO is the standard for S3-compatible storage in private or hybrid clouds.

What happens if my backup bucket is corrupted? That's the nightmare scenario. We mitigate this by using MinIO's built-in bucket replication features to sync our backups to a secondary, geographically distant cluster.

Final Thoughts

We’re currently still experimenting with the best retention policies. While we’ve settled on a 30-day retention window, I’m not entirely sure how this will scale once we move from 50 persistent volumes to 500. We’ll likely need to implement automated lifecycle management within MinIO to prune old snapshots, but for now, we’re manually monitoring the bucket size. We're also considering moving to an immutable storage backend for the backups to prevent accidental deletion, but that’s a task for the next sprint.