{"page":{"agent_metadata":{"content_type":"guide","outputs":["pv-snapshot-strategy","application-consistent-backup-setup","cross-cluster-replication-config","database-operator-dr-procedures","restore-ordering-plan"],"prerequisites":["kubernetes-statefulsets","persistent-volumes","csi-basics","database-operations"]},"categories":["kubernetes"],"content_plain":"Stateful Workload Disaster Recovery# Stateless workloads are easy to recover \u0026ndash; redeploy from Git and they are running. Stateful workloads carry data that cannot be regenerated. Databases, message queues, object stores, and anything with a PersistentVolume needs a deliberate DR strategy that goes beyond \u0026ldquo;we have Velero.\u0026rdquo;\nThe fundamental challenge: you must capture data at a point in time where the application state is consistent, replicate that data to a recovery site, and restore it in the correct order. Get any of these wrong and you recover corrupted data or a broken dependency chain.\nPersistentVolume Snapshot Strategies# CSI VolumeSnapshots# CSI snapshots are the Kubernetes-native way to snapshot PVs. They work with any CSI driver that supports the snapshot feature (EBS CSI, GCE PD CSI, Azure Disk CSI, Longhorn, Ceph).\n# First, create a VolumeSnapshotClass apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-snapclass driver: ebs.csi.aws.com # Match your CSI driver deletionPolicy: Retain # Keep snapshot when VolumeSnapshot object is deleted --- # Take a snapshot apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: postgres-data-snap-20260222 namespace: production spec: volumeSnapshotClassName: csi-snapclass source: persistentVolumeClaimName: data-postgres-0Restore from a snapshot by creating a new PVC that references it:\napiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-postgres-0-restored namespace: production spec: accessModes: [\u0026#34;ReadWriteOnce\u0026#34;] storageClassName: gp3 resources: requests: storage: 100Gi dataSource: name: postgres-data-snap-20260222 kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.ioCloud-Provider Snapshots# Cloud snapshots happen at the block storage level. They are faster than file-level backups and can be copied cross-region for DR.\n# AWS: snapshot an EBS volume and copy to another region SNAP_ID=$(aws ec2 create-snapshot --volume-id vol-0abc123 --description \u0026#34;postgres DR\u0026#34; --query SnapshotId --output text) aws ec2 copy-snapshot --source-region us-east-1 --source-snapshot-id $SNAP_ID --destination-region eu-west-1Automate cross-region snapshot copies with AWS DLM (Data Lifecycle Manager) or equivalent. Without automation, cross-region copies are forgotten within weeks.\nApplication-Consistent vs Crash-Consistent Backups# This is the distinction that matters most for databases.\nCrash-consistent: A snapshot taken at an arbitrary point in time. The volume captures whatever was on disk at that instant, including half-written pages and uncommitted transactions. This is what you get from a raw CSI snapshot or cloud volume snapshot while the database is running.\nApplication-consistent: The application is quiesced before the snapshot. For databases, this means flushing dirty pages to disk, checkpointing the WAL, and ensuring the data directory is in a recoverable state.\nA crash-consistent snapshot of PostgreSQL will usually recover \u0026ndash; PostgreSQL replays the WAL on startup. But \u0026ldquo;usually\u0026rdquo; is not good enough for production DR. Some databases (MySQL with MyISAM tables, older MongoDB) can produce unrecoverable snapshots from crash-consistent backups.\nPostgreSQL Application-Consistent Snapshot# # Freeze writes, take snapshot, thaw kubectl exec -n production postgres-0 -- psql -c \u0026#34;SELECT pg_backup_start(\u0026#39;dr-snapshot\u0026#39;);\u0026#34; # Take the CSI snapshot while writes are frozen kubectl apply -f volume-snapshot.yaml # Thaw writes kubectl exec -n production postgres-0 -- psql -c \u0026#34;SELECT pg_backup_stop();\u0026#34;Velero Pre/Post Backup Hooks# Velero supports hooks that run commands in pods before and after backup:\napiVersion: v1 kind: Pod metadata: annotations: pre.hook.backup.velero.io/container: postgres pre.hook.backup.velero.io/command: \u0026#39;[\u0026#34;/bin/bash\u0026#34;, \u0026#34;-c\u0026#34;, \u0026#34;pg_backup_start(\u0026#39;\u0026#39;velero\u0026#39;\u0026#39;)\u0026#34;]\u0026#39; post.hook.backup.velero.io/container: postgres post.hook.backup.velero.io/command: \u0026#39;[\u0026#34;/bin/bash\u0026#34;, \u0026#34;-c\u0026#34;, \u0026#34;pg_backup_stop()\u0026#34;]\u0026#39;For MySQL:\npre.hook.backup.velero.io/command: \u0026#39;[\u0026#34;/bin/bash\u0026#34;, \u0026#34;-c\u0026#34;, \u0026#34;mysql -u root -e \\\u0026#34;FLUSH TABLES WITH READ LOCK;\\\u0026#34;\u0026#34;]\u0026#39; post.hook.backup.velero.io/command: \u0026#39;[\u0026#34;/bin/bash\u0026#34;, \u0026#34;-c\u0026#34;, \u0026#34;mysql -u root -e \\\u0026#34;UNLOCK TABLES;\\\u0026#34;\u0026#34;]\u0026#39;Storage Replication for Cross-Cluster DR# Portworx# Portworx supports synchronous and asynchronous replication between clusters. Asynchronous replication (PX-DR) sends incremental snapshots to a remote cluster on a schedule.\n# Create a replication schedule storkctl create migration-schedule postgres-dr \\ --cluster-pair remote-cluster \\ --namespaces production \\ --interval 15 # minutesPortworx also supports synchronous replication (PX-Metro) for zero RPO, but this requires low-latency (\u0026lt;10ms) connections between sites \u0026ndash; essentially the same data center or metro area.\nLonghorn# Longhorn supports DR volumes that replicate to an S3-compatible backup target. The secondary cluster mounts the DR volume in standby mode.\napiVersion: longhorn.io/v1beta2 kind: Volume metadata: name: postgres-data spec: numberOfReplicas: 3 recurringJobs: - name: backup-every-15m task: backup cron: \u0026#34;*/15 * * * *\u0026#34; retain: 10 labels: type: drOn the DR cluster, create a DR volume pointing to the same backup target:\napiVersion: longhorn.io/v1beta2 kind: Volume metadata: name: postgres-data-dr spec: fromBackup: \u0026#34;s3://longhorn-backups@us-east-1/backups/postgres-data\u0026#34; standby: trueTo activate: set standby: false and attach the volume. Longhorn replays the latest backup and the volume becomes read-write.\nRook-Ceph Cross-Cluster# Rook-Ceph supports RBD mirroring between two Ceph clusters. This provides block-level replication for PVs backed by Ceph.\napiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: replicated-pool spec: replicated: size: 3 mirroring: enabled: true mode: image snapshotSchedules: - interval: 5mBootstrap the mirror peer between clusters, and Ceph replicates RBD images asynchronously. RPO depends on the snapshot schedule interval.\nDatabase Operator DR# Modern database operators handle DR natively. Use them instead of building custom snapshot pipelines.\nCloudNativePG (PostgreSQL)# CloudNativePG supports continuous backup to object storage and point-in-time recovery (PITR):\napiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: production-pg spec: instances: 3 backup: barmanObjectStore: destinationPath: s3://pg-backups/production s3Credentials: accessKeyID: name: aws-creds key: ACCESS_KEY_ID secretAccessKey: name: aws-creds key: SECRET_ACCESS_KEY retentionPolicy: \u0026#34;30d\u0026#34; scheduledBackups: - name: daily schedule: \u0026#34;0 2 * * *\u0026#34; backupOwnerReference: selfRestore to a DR cluster by creating a new Cluster resource pointing to the backup location:\napiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: production-pg-restored spec: instances: 3 bootstrap: recovery: source: production-pg recoveryTarget: targetTime: \u0026#34;2026-02-22T06:00:00Z\u0026#34; # Point-in-time externalClusters: - name: production-pg barmanObjectStore: destinationPath: s3://pg-backups/production s3Credentials: accessKeyID: { name: aws-creds, key: ACCESS_KEY_ID } secretAccessKey: { name: aws-creds, key: SECRET_ACCESS_KEY }Percona Operator (MySQL/MongoDB)# Percona XtraDB Cluster Operator supports scheduled backups to S3:\napiVersion: pxc.percona.com/v1 kind: PerconaXtraDBClusterBackup metadata: name: daily-backup spec: pxcCluster: production-mysql storageName: s3-backupMongoDB Community Operator# The MongoDB Community Operator does not include built-in backup CRDs. Use Percona Backup for MongoDB (PBM) as a sidecar or external tool for consistent backups with PITR support.\nMessage Queue DR# Kafka MirrorMaker 2# MirrorMaker 2 replicates topics between Kafka clusters. Deploy it as a KafkaConnect resource with Strimzi:\napiVersion: kafka.strimzi.io/v1beta2 kind: KafkaMirrorMaker2 metadata: name: dr-mirror spec: version: 3.7.0 replicas: 3 connectCluster: dr-cluster clusters: - alias: primary bootstrapServers: primary-kafka-bootstrap:9092 - alias: dr-cluster bootstrapServers: dr-kafka-bootstrap:9092 mirrors: - sourceCluster: primary targetCluster: dr-cluster topicsPattern: \u0026#34;.*\u0026#34; groupsPattern: \u0026#34;.*\u0026#34;MirrorMaker replicates topic data, consumer group offsets, and ACLs. RPO depends on replication lag, typically seconds.\nRabbitMQ Shovel# Shovel moves messages from a queue on one broker to a queue on another. Configure it as a policy for DR:\nrabbitmqctl set_parameter shovel dr-orders \\ \u0026#39;{\u0026#34;src-protocol\u0026#34;: \u0026#34;amqp091\u0026#34;, \u0026#34;src-uri\u0026#34;: \u0026#34;amqp://primary:5672\u0026#34;, \u0026#34;src-queue\u0026#34;: \u0026#34;orders\u0026#34;, \u0026#34;dest-protocol\u0026#34;: \u0026#34;amqp091\u0026#34;, \u0026#34;dest-uri\u0026#34;: \u0026#34;amqp://dr-site:5672\u0026#34;, \u0026#34;dest-queue\u0026#34;: \u0026#34;orders\u0026#34;}\u0026#39;Shovel is point-to-point. For full cluster replication, use Federation or configure Shovel for each critical queue.\nThe Ordering Problem# This is where most DR recoveries fail in practice. Kubernetes resources have dependencies, and restoring them in the wrong order produces errors that cascade.\nThe correct restore order:\nNamespaces and RBAC \u0026ndash; everything depends on namespaces existing CRDs \u0026ndash; operators need their CRDs before they can reconcile Operators \u0026ndash; install and wait for them to be ready Storage \u0026ndash; PVCs, restore PV snapshots, wait for volumes to bind Databases \u0026ndash; restore data, wait for them to become ready Message queues \u0026ndash; restore data, wait for cluster formation Application workloads \u0026ndash; deploy services that depend on databases and queues Ingress and DNS \u0026ndash; only route traffic once everything is healthy Velero restores in a defined order (namespaces, then CRDs, then cluster-scoped, then namespaced), but it does not wait for readiness between steps. A database pod may be \u0026ldquo;restored\u0026rdquo; (the Pod object exists) but not yet accepting connections when the application pods start trying to connect.\nHandle this with init containers that check dependencies:\ninitContainers: - name: wait-for-postgres image: busybox:1.36 command: [\u0026#39;sh\u0026#39;, \u0026#39;-c\u0026#39;, \u0026#39;until nc -z postgres.production.svc 5432; do echo waiting; sleep 5; done\u0026#39;]Or use Kubernetes startup probes with generous timeouts for applications that connect to databases on startup. The application will crashloop until the database is ready, and Kubernetes will keep restarting it \u0026ndash; this is ugly but functional.\nThe better approach: restore infrastructure (storage, databases, queues) first, validate health, then restore application workloads in a second pass. Two-phase restore is more work to automate but significantly more reliable than hoping everything comes up in the right order.\n","date":"2026-02-22","description":"DR strategies for stateful Kubernetes workloads: CSI and cloud volume snapshots, application-consistent vs crash-consistent backups, cross-cluster storage replication, database and message queue operator DR, and the critical ordering problem during restore.","lastmod":"2026-02-22","levels":["intermediate","advanced"],"reading_time_minutes":7,"section":"knowledge","skills":["pv-snapshot-management","application-consistent-backup","cross-cluster-replication","database-operator-dr","restore-ordering"],"tags":["disaster-recovery","stateful-workloads","persistent-volumes","csi-snapshots","portworx","longhorn","rook-ceph","cloudnativepg","percona-operator","kafka","rabbitmq"],"title":"Stateful Workload Disaster Recovery: Storage Replication, Database Operators, and Restore Ordering","tools":["kubectl","velero","helm","pg_basebackup","etcdctl"],"url":"https://agent-zone.ai/knowledge/kubernetes/stateful-workload-dr/","word_count":1298}}