How do distributed databases handle consistency during a network failure?

Distributed databases handle consistency during network failures by balancing data reliability with system availability, often relying on predefined consistency models and replication strategies. When a network partition occurs (nodes can’t communicate), these systems face a trade-off: prioritize immediate consistency by blocking operations until nodes reconnect or allow temporary inconsistencies to maintain availability. The approach depends on the database’s design goals and the consistency model it implements, such as strong consistency, eventual consistency, or a tunable middle ground.

In systems prioritizing strong consistency (like Google Spanner or etcd), writes require confirmation from a majority of nodes using mechanisms like quorum-based replication or consensus algorithms (e.g., Raft). During a network failure, if a node loses contact with the majority, it becomes unable to process writes, ensuring no conflicting updates occur. For example, in a 5-node cluster using Raft, if 3 nodes form a majority partition, only those 3 can accept writes. The remaining 2 nodes become read-only until the network heals. This prevents split-brain scenarios but sacrifices availability for partitions lacking a majority. Tools like leader election and heartbeat mechanisms detect failures and enforce these rules.

For databases favoring availability (like Apache Cassandra or DynamoDB), eventual consistency allows temporary divergence. During a partition, writes can proceed on both sides of the split, creating conflicting versions. Once connectivity restores, conflict resolution mechanisms reconcile differences. For example, Cassandra uses “last write wins” (based on timestamps) or application-defined logic to merge data. Amazon DynamoDB uses vector clocks to track version history, letting applications resolve conflicts manually. Some systems also employ Conflict-Free Replicated Data Types (CRDTs), which automatically merge updates (e.g., incrementing counters) without manual intervention. These approaches keep the system operational during outages but require careful handling of stale or conflicting data post-recovery.