Kubernetes disaster recovery: backup, restore and the etcd snapshot you forgot

Kubernetes disaster recovery is not complicated. It is a checklist, and most teams get three-quarters of the way down it and stop. The missing quarter is the one that matters in an actual disaster.

This is the full checklist, from etcd snapshots to the restore runbook you need to test before you need to run it.

What you need to back up

Not everything on a cluster is worth restoring. Configuration declared in Git is restored by reapplying it. The things that are not in Git — and cannot be regenerated — are the backup surface.

1. etcd

etcd is the cluster's source of truth. Lose it and you lose every Kubernetes object: pods, deployments, services, secrets, configmaps, CRDs and their instances.

Back it up with etcdctl:

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

On managed Kubernetes (EKS, AKS, GKE), the control plane is the provider's problem and etcd snapshots are not accessible. On self-managed clusters (RKE2, kubeadm), you are the control plane owner and this is your job.

For RKE2, the built-in snapshot mechanism is simpler:

# Take a snapshot
rke2 etcd-snapshot save --s3 \
  --s3-bucket=my-backups \
  --s3-region=us-east-1 \
  --s3-folder=rke2-etcd

# List snapshots
rke2 etcd-snapshot ls --s3

Configure automatic snapshots with a cron schedule in the RKE2 config. A cluster that runs without automated etcd snapshots is a cluster one kubectl delete away from unrecoverable.

2. Persistent volumes

StatefulSets and PVC-backed pods store data outside etcd. The etcd snapshot captures the PersistentVolumeClaim object and the pod spec; it does not capture the data on the volume.

Back up volumes at the storage layer, not the Kubernetes layer:

• AWS EBS: snapshots via AWS Backup or a CronJob that calls the AWS API
• Azure Disks: snapshots via Azure Backup
• On-prem / NFS / Ceph: Velero with the appropriate storage plugin

Velero is the standard tool for Kubernetes-native backup and restore. It backs up both Kubernetes objects and persistent volumes to an S3-compatible object store:

velero install \
  --provider aws \
  --bucket my-k8s-backups \
  --backup-location-config region=us-east-1 \
  --snapshot-location-config region=us-east-1 \
  --use-volume-snapshots=true

A scheduled backup that runs nightly and retains 30 days:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily
  namespace: velero
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces:
      - "*"
    excludedNamespaces:
      - velero
      - kube-system
    ttl: 720h  # 30 days

3. Secrets that are not in External Secrets Operator

If you use External Secrets Operator or a similar tool that syncs secrets from an external store (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault), the secrets themselves are backed up with the external store and do not need a separate process.

If you have Kubernetes Secrets created directly — and you should not, but if you do — an etcd snapshot covers them. That is also the argument for not creating them directly: it means etcd contains secret material, and the etcd backup inherits that classification.

What you do not need to back up

Skip	Why
Pods, Deployments, Services	Recreated by reapplying the YAML from Git
ConfigMaps	Same — reapplied from Git, and the etcd snapshot covers them anyway
Cert-manager certificates	Reissued automatically on restore. Backing them up doubles your certificate management state for no benefit
Logs and metrics	Not cluster state. If you need historical logs, back them up at the log aggregation layer
Container images	Stored in a registry, not the cluster. If the registry is yours, back up the registry

The restore that fails

The most common restore failure is not a technical one. It is that the restore procedure was written but never tested, and the first time anyone runs it is at 3am during an incident.

A restore procedure that is known to work looks like this:

# 1. Restore etcd from snapshot
rke2 etcd-snapshot-restore \
  --s3 \
  --s3-bucket=my-backups \
  --s3-region=us-east-1 \
  --s3-folder=rke2-etcd \
  --name=etcd-20260927-020000

# 2. Verify cluster is healthy
kubectl get nodes
kubectl get pods --all-namespaces | grep -v Running

# 3. Restore persistent volumes from Velero
velero restore create --from-backup daily-20260927-020000

# 4. Verify PVCs are bound
kubectl get pvc --all-namespaces | grep -v Bound

# 5. Verify application health
# App-specific — health endpoints, integration tests, smoke tests

The step that breaks is step 4. A PVC that was Bound before the disaster may be Pending after a restore because the StorageClass is missing, the CSI driver is not installed or the volume snapshot provisioner is not running. These are not restore problems — they are bootstrap problems, and they can only be caught by testing the restore.

Testing restores without destroying production

Run a restore test in a separate cluster. Spin up a temporary cluster, restore the latest etcd snapshot and Velero backup, and assert that the control plane is healthy and the application endpoints respond.

This test costs the price of a temporary cluster for a few hours. The alternative — discovering the restore is broken during an incident — costs the incident plus the time to fix the restore procedure plus the downtime the backup was supposed to prevent.

Schedule it. A restore test that runs quarterly is worth more than a backup that runs nightly and has never been restored.

---

Hitting a DR gap or want a restore runbook tested? I take on Kubernetes platform builds and fractional DevOps retainers as freelance and contract work — independently, for teams in the US, EU and APAC.