--- title: "Data Consistency in Multi-Region Deployments" description: "Managing data consistency across regions — the CAP theorem in practice, conflict resolution strategies (last-write-wins, vector clocks, CRDTs), eventual consistency patterns, strong consistency with CockroachDB and Spanner, and real-world consistency bugs with prevention strategies." url: https://agent-zone.ai/knowledge/databases/data-consistency-multi-region/ section: knowledge date: 2026-02-22 categories: ["databases"] tags: ["consistency","multi-region","distributed-systems","cap-theorem","crdt","eventual-consistency","cockroachdb","spanner","conflict-resolution","vector-clocks"] skills: ["distributed-systems-design","consistency-model-selection","conflict-resolution-design","multi-region-architecture"] tools: ["cockroachdb","spanner","dynamodb","cosmos-db","cassandra"] levels: ["intermediate","advanced"] word_count: 1135 formats: json: https://agent-zone.ai/knowledge/databases/data-consistency-multi-region/index.json html: https://agent-zone.ai/knowledge/databases/data-consistency-multi-region/?format=html api: https://api.agent-zone.ai/api/v1/knowledge/search?q=Data+Consistency+in+Multi-Region+Deployments --- # Data Consistency in Multi-Region Deployments When you replicate data across regions, you are forced to choose between consistency, latency, and availability. You cannot have all three. Every multi-region system makes this tradeoff explicitly or, more dangerously, implicitly by ignoring it until production exposes the consequences. ## The Fundamental Tension **Strong consistency** means every read sees the most recent write, regardless of which region it comes from. This requires cross-region coordination on every write (30-100ms per round trip). **Eventual consistency** means reads might see stale data, but replicas converge given enough time -- usually milliseconds to seconds, but during partitions it can be minutes. The practical question is not "which is better?" but "which data needs which guarantee?" Most applications have a mix: financial balances need strong consistency, user profile pictures can be eventually consistent, shopping carts need something in between. ## Conflict Resolution Strategies When two regions accept conflicting writes to the same data, something must decide which write wins. There are four main approaches. ### Last-Write-Wins (LWW) The write with the latest timestamp wins. Simple, but dangerous. DynamoDB Global Tables and Cosmos DB both default to LWW. The problem: clocks across regions are only synchronized within a few milliseconds (NTP), so two "simultaneous" writes can have arbitrary ordering. A user updates their email in US-East while a background job touches their profile in US-West 2ms later -- the background job wins, silently losing the email update. LWW works when overwrites are acceptable (analytics counters, device locations, session data). It does not work when every write must be preserved. ### Vector Clocks Each node maintains a vector of logical clocks tracking which writes it has seen from every other node. When two writes conflict, the vector clocks reveal whether one causally depends on the other (no conflict) or they are truly concurrent (real conflict). Amazon's original Dynamo paper used vector clocks, but DynamoDB (the service) switched to LWW because most applications could not correctly implement the merge logic that true concurrency requires. ### CRDTs (Conflict-Free Replicated Data Types) CRDTs are data structures where concurrent modifications always merge automatically. Common types: **G-Counter** (grow-only, used for page views), **PN-Counter** (increment/decrement, used for inventory), **OR-Set** (add/remove set, used for shopping carts), **LWW-Register** (single value with timestamp). Redis Enterprise uses CRDTs for active-active geo-replication. The tradeoff: CRDTs restrict what operations you can perform. You cannot build a general-purpose CRDT for arbitrary SQL UPDATE statements. ### Application-Level Merge The application receives both conflicting versions and decides how to merge them. A collaborative document editor, for example, uses operational transformation to merge concurrent edits rather than picking one. This is the right approach when the conflict resolution logic is domain-specific and cannot be captured by a generic strategy. ## Eventual Consistency Patterns That Work These patterns are specifically designed to be safe under eventual consistency: **Shopping carts:** Model the cart as a set of add/remove operations rather than a final state. Concurrent adds in different regions merge cleanly. **Session data:** Store sessions in the nearest region. Worst case on region switch is re-authentication. Use TTLs so stale sessions expire. **Analytics and counters:** Per-region counters aggregated on read. Momentarily inaccurate by a few counts, which is fine for dashboards. ## Strong Consistency Patterns That Work ### Single-Region Write, Multi-Region Read Route all writes to one region. Replicate asynchronously to other regions for reads. This gives strong consistency for writes and eventual consistency for reads. It works for 90% of applications. The downside: write traffic goes cross-region for users far from the primary, adding 30-100ms to write latency. ### CockroachDB -- Strongly Consistent Multi-Region CockroachDB uses the Raft consensus protocol to maintain strong consistency across regions. You control where data lives using table localities: ```sql -- Pin frequently-read data to a specific region for fast local reads ALTER TABLE users SET LOCALITY REGIONAL BY ROW; -- Data that is read globally but written from one region ALTER TABLE products SET LOCALITY GLOBAL; -- Data that stays in the region where it was written ALTER TABLE orders SET LOCALITY REGIONAL BY ROW; ``` `REGIONAL BY ROW` stores each row in the region of its `crdb_region` column -- fast local reads, but writes require Raft consensus across regions (50-150ms). `GLOBAL` tables make reads fast everywhere but writes slower. ### Google Spanner -- TrueTime Spanner uses GPS-synchronized atomic clocks (TrueTime) to assign globally consistent timestamps to transactions without cross-region coordination for reads. This is hardware-level infrastructure that no one else has replicated. **Real-world Spanner latency:** Single-region writes complete in 5-10ms. Multi-region writes that span regions take 50-200ms depending on the regions involved. Stale reads (reading data that is at least 10 seconds old) are fast from any region. ## Real-World Consistency Bugs ### Bug: Double-Spend from Replication Lag A payment service checks the user's balance in the local region, deducts the amount, and writes the new balance. The user makes two purchases simultaneously from different regions. Both reads see the full balance. Both deductions succeed. The user spent their money twice. **Prevention:** Route all financial writes through a single region. Use a distributed lock or serializable isolation if the database supports it. Or use a system like CockroachDB where cross-region writes are serializable by default. ### Bug: Phantom Reads in Inventory Service A reads inventory count in US-East (shows 1 remaining). Service B reads inventory count in US-West (also shows 1 remaining). Both reserve the item. You have now sold an item you do not have. **Prevention:** Use a conditional write. Instead of "read count, then write new count," use "UPDATE inventory SET count = count - 1 WHERE count > 0 AND id = X" and verify the affected row count. This works even with eventual consistency as long as each region has its own inventory partition, or all inventory writes go through a single primary. ### Bug: Stale Cache After Failover Application caches frequently-read data locally. Primary region fails. Application fails over to DR region. The DR region's database is a few seconds behind. The application cache was built from the primary's data. Now the cache is ahead of the database, causing the application to show data that the database claims does not exist yet. **Prevention:** Invalidate all caches during failover. Accept the performance hit of cold caches in the DR region. Build cache warming into your failover runbook. ## Choosing Your Consistency Model | Data Type | Consistency Needed | Recommended Approach | |---|---|---| | Financial balances | Strong | Single-region write or CockroachDB/Spanner | | User profiles | Eventual (seconds) | Async replication, LWW for conflicts | | Shopping carts | Convergent | CRDTs or operation-based merge | | Inventory counts | Strong for writes | Conditional writes, single-region primary | | Analytics/counters | Eventual (minutes OK) | Per-region counters, aggregate on read | | Session data | Eventual | Region-local with TTL | | Audit logs | Append-only, eventual | Per-region append, async replication | Start with the strongest consistency you can afford for each data type, then relax it only when latency requirements demand it. It is far easier to relax consistency than to strengthen it after the system is in production.