--- title: "Stateful Workload Disaster Recovery: Storage Replication, Database Operators, and Restore Ordering" description: "DR strategies for stateful Kubernetes workloads: CSI and cloud volume snapshots, application-consistent vs crash-consistent backups, cross-cluster storage replication, database and message queue operator DR, and the critical ordering problem during restore." url: https://agent-zone.ai/knowledge/kubernetes/stateful-workload-dr/ section: knowledge date: 2026-02-22 categories: ["kubernetes"] tags: ["disaster-recovery","stateful-workloads","persistent-volumes","csi-snapshots","portworx","longhorn","rook-ceph","cloudnativepg","percona-operator","kafka","rabbitmq"] skills: ["pv-snapshot-management","application-consistent-backup","cross-cluster-replication","database-operator-dr","restore-ordering"] tools: ["kubectl","velero","helm","pg_basebackup","etcdctl"] levels: ["intermediate","advanced"] word_count: 1298 formats: json: https://agent-zone.ai/knowledge/kubernetes/stateful-workload-dr/index.json html: https://agent-zone.ai/knowledge/kubernetes/stateful-workload-dr/?format=html api: https://api.agent-zone.ai/api/v1/knowledge/search?q=Stateful+Workload+Disaster+Recovery%3A+Storage+Replication%2C+Database+Operators%2C+and+Restore+Ordering --- # Stateful Workload Disaster Recovery Stateless workloads are easy to recover -- redeploy from Git and they are running. Stateful workloads carry data that cannot be regenerated. Databases, message queues, object stores, and anything with a PersistentVolume needs a deliberate DR strategy that goes beyond "we have Velero." The fundamental challenge: you must capture data at a point in time where the application state is consistent, replicate that data to a recovery site, and restore it in the correct order. Get any of these wrong and you recover corrupted data or a broken dependency chain. ## PersistentVolume Snapshot Strategies ### CSI VolumeSnapshots CSI snapshots are the Kubernetes-native way to snapshot PVs. They work with any CSI driver that supports the snapshot feature (EBS CSI, GCE PD CSI, Azure Disk CSI, Longhorn, Ceph). ```yaml # First, create a VolumeSnapshotClass apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-snapclass driver: ebs.csi.aws.com # Match your CSI driver deletionPolicy: Retain # Keep snapshot when VolumeSnapshot object is deleted --- # Take a snapshot apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: postgres-data-snap-20260222 namespace: production spec: volumeSnapshotClassName: csi-snapclass source: persistentVolumeClaimName: data-postgres-0 ``` Restore from a snapshot by creating a new PVC that references it: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-postgres-0-restored namespace: production spec: accessModes: ["ReadWriteOnce"] storageClassName: gp3 resources: requests: storage: 100Gi dataSource: name: postgres-data-snap-20260222 kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io ``` ### Cloud-Provider Snapshots Cloud snapshots happen at the block storage level. They are faster than file-level backups and can be copied cross-region for DR. ```bash # AWS: snapshot an EBS volume and copy to another region SNAP_ID=$(aws ec2 create-snapshot --volume-id vol-0abc123 --description "postgres DR" --query SnapshotId --output text) aws ec2 copy-snapshot --source-region us-east-1 --source-snapshot-id $SNAP_ID --destination-region eu-west-1 ``` Automate cross-region snapshot copies with AWS DLM (Data Lifecycle Manager) or equivalent. Without automation, cross-region copies are forgotten within weeks. ## Application-Consistent vs Crash-Consistent Backups This is the distinction that matters most for databases. **Crash-consistent:** A snapshot taken at an arbitrary point in time. The volume captures whatever was on disk at that instant, including half-written pages and uncommitted transactions. This is what you get from a raw CSI snapshot or cloud volume snapshot while the database is running. **Application-consistent:** The application is quiesced before the snapshot. For databases, this means flushing dirty pages to disk, checkpointing the WAL, and ensuring the data directory is in a recoverable state. A crash-consistent snapshot of PostgreSQL will usually recover -- PostgreSQL replays the WAL on startup. But "usually" is not good enough for production DR. Some databases (MySQL with MyISAM tables, older MongoDB) can produce unrecoverable snapshots from crash-consistent backups. ### PostgreSQL Application-Consistent Snapshot ```bash # Freeze writes, take snapshot, thaw kubectl exec -n production postgres-0 -- psql -c "SELECT pg_backup_start('dr-snapshot');" # Take the CSI snapshot while writes are frozen kubectl apply -f volume-snapshot.yaml # Thaw writes kubectl exec -n production postgres-0 -- psql -c "SELECT pg_backup_stop();" ``` ### Velero Pre/Post Backup Hooks Velero supports hooks that run commands in pods before and after backup: ```yaml apiVersion: v1 kind: Pod metadata: annotations: pre.hook.backup.velero.io/container: postgres pre.hook.backup.velero.io/command: '["/bin/bash", "-c", "pg_backup_start(''velero'')"]' post.hook.backup.velero.io/container: postgres post.hook.backup.velero.io/command: '["/bin/bash", "-c", "pg_backup_stop()"]' ``` For MySQL: ```yaml pre.hook.backup.velero.io/command: '["/bin/bash", "-c", "mysql -u root -e \"FLUSH TABLES WITH READ LOCK;\""]' post.hook.backup.velero.io/command: '["/bin/bash", "-c", "mysql -u root -e \"UNLOCK TABLES;\""]' ``` ## Storage Replication for Cross-Cluster DR ### Portworx Portworx supports synchronous and asynchronous replication between clusters. Asynchronous replication (PX-DR) sends incremental snapshots to a remote cluster on a schedule. ```bash # Create a replication schedule storkctl create migration-schedule postgres-dr \ --cluster-pair remote-cluster \ --namespaces production \ --interval 15 # minutes ``` Portworx also supports synchronous replication (PX-Metro) for zero RPO, but this requires low-latency (<10ms) connections between sites -- essentially the same data center or metro area. ### Longhorn Longhorn supports DR volumes that replicate to an S3-compatible backup target. The secondary cluster mounts the DR volume in standby mode. ```yaml apiVersion: longhorn.io/v1beta2 kind: Volume metadata: name: postgres-data spec: numberOfReplicas: 3 recurringJobs: - name: backup-every-15m task: backup cron: "*/15 * * * *" retain: 10 labels: type: dr ``` On the DR cluster, create a DR volume pointing to the same backup target: ```yaml apiVersion: longhorn.io/v1beta2 kind: Volume metadata: name: postgres-data-dr spec: fromBackup: "s3://longhorn-backups@us-east-1/backups/postgres-data" standby: true ``` To activate: set `standby: false` and attach the volume. Longhorn replays the latest backup and the volume becomes read-write. ### Rook-Ceph Cross-Cluster Rook-Ceph supports RBD mirroring between two Ceph clusters. This provides block-level replication for PVs backed by Ceph. ```yaml apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: replicated-pool spec: replicated: size: 3 mirroring: enabled: true mode: image snapshotSchedules: - interval: 5m ``` Bootstrap the mirror peer between clusters, and Ceph replicates RBD images asynchronously. RPO depends on the snapshot schedule interval. ## Database Operator DR Modern database operators handle DR natively. Use them instead of building custom snapshot pipelines. ### CloudNativePG (PostgreSQL) CloudNativePG supports continuous backup to object storage and point-in-time recovery (PITR): ```yaml apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: production-pg spec: instances: 3 backup: barmanObjectStore: destinationPath: s3://pg-backups/production s3Credentials: accessKeyID: name: aws-creds key: ACCESS_KEY_ID secretAccessKey: name: aws-creds key: SECRET_ACCESS_KEY retentionPolicy: "30d" scheduledBackups: - name: daily schedule: "0 2 * * *" backupOwnerReference: self ``` Restore to a DR cluster by creating a new Cluster resource pointing to the backup location: ```yaml apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: production-pg-restored spec: instances: 3 bootstrap: recovery: source: production-pg recoveryTarget: targetTime: "2026-02-22T06:00:00Z" # Point-in-time externalClusters: - name: production-pg barmanObjectStore: destinationPath: s3://pg-backups/production s3Credentials: accessKeyID: { name: aws-creds, key: ACCESS_KEY_ID } secretAccessKey: { name: aws-creds, key: SECRET_ACCESS_KEY } ``` ### Percona Operator (MySQL/MongoDB) Percona XtraDB Cluster Operator supports scheduled backups to S3: ```yaml apiVersion: pxc.percona.com/v1 kind: PerconaXtraDBClusterBackup metadata: name: daily-backup spec: pxcCluster: production-mysql storageName: s3-backup ``` ### MongoDB Community Operator The MongoDB Community Operator does not include built-in backup CRDs. Use Percona Backup for MongoDB (PBM) as a sidecar or external tool for consistent backups with PITR support. ## Message Queue DR ### Kafka MirrorMaker 2 MirrorMaker 2 replicates topics between Kafka clusters. Deploy it as a KafkaConnect resource with Strimzi: ```yaml apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaMirrorMaker2 metadata: name: dr-mirror spec: version: 3.7.0 replicas: 3 connectCluster: dr-cluster clusters: - alias: primary bootstrapServers: primary-kafka-bootstrap:9092 - alias: dr-cluster bootstrapServers: dr-kafka-bootstrap:9092 mirrors: - sourceCluster: primary targetCluster: dr-cluster topicsPattern: ".*" groupsPattern: ".*" ``` MirrorMaker replicates topic data, consumer group offsets, and ACLs. RPO depends on replication lag, typically seconds. ### RabbitMQ Shovel Shovel moves messages from a queue on one broker to a queue on another. Configure it as a policy for DR: ```bash rabbitmqctl set_parameter shovel dr-orders \ '{"src-protocol": "amqp091", "src-uri": "amqp://primary:5672", "src-queue": "orders", "dest-protocol": "amqp091", "dest-uri": "amqp://dr-site:5672", "dest-queue": "orders"}' ``` Shovel is point-to-point. For full cluster replication, use Federation or configure Shovel for each critical queue. ## The Ordering Problem This is where most DR recoveries fail in practice. Kubernetes resources have dependencies, and restoring them in the wrong order produces errors that cascade. The correct restore order: 1. **Namespaces and RBAC** -- everything depends on namespaces existing 2. **CRDs** -- operators need their CRDs before they can reconcile 3. **Operators** -- install and wait for them to be ready 4. **Storage** -- PVCs, restore PV snapshots, wait for volumes to bind 5. **Databases** -- restore data, wait for them to become ready 6. **Message queues** -- restore data, wait for cluster formation 7. **Application workloads** -- deploy services that depend on databases and queues 8. **Ingress and DNS** -- only route traffic once everything is healthy Velero restores in a defined order (namespaces, then CRDs, then cluster-scoped, then namespaced), but it does not wait for readiness between steps. A database pod may be "restored" (the Pod object exists) but not yet accepting connections when the application pods start trying to connect. Handle this with init containers that check dependencies: ```yaml initContainers: - name: wait-for-postgres image: busybox:1.36 command: ['sh', '-c', 'until nc -z postgres.production.svc 5432; do echo waiting; sleep 5; done'] ``` Or use Kubernetes startup probes with generous timeouts for applications that connect to databases on startup. The application will crashloop until the database is ready, and Kubernetes will keep restarting it -- this is ugly but functional. The better approach: restore infrastructure (storage, databases, queues) first, validate health, then restore application workloads in a second pass. Two-phase restore is more work to automate but significantly more reliable than hoping everything comes up in the right order.