I\u2019ll also share a real-world experience with GlusterFS split-brain and the lessons learned.<\/p>\n
- \n
- Split-Brain Scenario in Distributed Systems<\/li>\n
- Quorum in Distributed Systems<\/li>\n
- How etcd avoids split brain scenario (Real World example)<\/li>\n<\/ol>\n\nIn a project, I faced a Split Brain Scenario<\/strong><\/b> in a GlusterFS NFS cluster<\/strong><\/b>, leading to data mismatches and a full restore from backup. It taught me to prioritize solutions that prevent such issues upfront<\/strong><\/b>.<\/div>\n<\/div>\n
Whenever we deploy distributed systems that deal with data, we usually deploy the nodes in different availability zones<\/strong> or different data centers (for on-prem environments) to ensure high availability.<\/p>\n
Example:<\/strong> databases like MongoDB, distributed storage systems like GlusterFS, and consensus-based clusters like etcd.<\/p>\n
Let\u2019s consider an example of an etcd cluster.<\/p>\n
Assume we have a 5-node etcd cluster<\/strong>. To maintain high availability, three nodes are deployed in Zone 1, while the remaining two are placed in Zone 2.<\/p>\n
![\"\"](\"https:\/\/blog.techiescamp.com\/content\/images\/2025\/02\/image-4.png\")
<\/figure>\nThis setup ensures that even if one zone goes down, there are still active nodes in another zone, maintaining service continuity.<\/p>\n
Split-Brain Scenario in Distributed Systems<\/h2>\n
In distributed systems, a split-brain scenario occurs when a group of nodes loses communication with each other, typically due to network partitioning.<\/p>\n
For example, in our 5-node etcd cluster, if network connectivity is lost between Zone 1 and Zone 2, the three nodes in Zone 1 may continue operating, forming a majority quorum.<\/p>\n
However, the two nodes in Zone 2 might also assume they should continue functioning independently, creating a second, conflicting cluster.<\/p>\n
![\"\"](\"https:\/\/blog.techiescamp.com\/content\/images\/2025\/02\/image-5.png\")
<\/figure>\nThis results in a classic split-brain problem<\/strong>, where:<\/p>\n
- \n
- Some nodes keep working and accept new data. Other nodes, cut off from the majority, also keep working, but they don\u2019t know what the other side is doing.<\/li>\n
- Since both sides are writing new data separately, they don\u2019t match anymore.<\/li>\n
- When the network issue is fixed, these nodes may struggle to catch up, leading to wrong or missing data.<\/li>\n<\/ul>\n
Quorum in Distributed Systems<\/h2>\n
To prevent Split-Brain scenarios, distributed systems use quorum-based decision-making<\/strong> to ensure consistency.<\/p>\n
Let us understand the concept of Quorum with examples.<\/p>\n
When you deploy database clusters, there are two key patterns:<\/p>\n
- \n
- Single-Primary Node<\/strong>: All writes go to the primary node, which then replicates those changes to secondary nodes.<\/li>\n
- Multi-Primary Node<\/strong>: In this setup, multiple nodes can accept writes simultaneously. When you write to any primary node, that node coordinates with others to ensure the write is properly replicated.<\/li>\n<\/ol>\n
So how do we choose the primary node?<\/p>\n
Most distributed databases use a leader election<\/strong> mechanism to determine the primary node. A classic example is etcd\u2019s Raft consensus algorithm<\/strong>.<\/p>\n
It works like a voting process<\/strong>: to elect a leader, a candidate node must receive a majority of votes<\/strong>.<\/p>\n
For example, in a 5-node system<\/strong>, at least 3 nodes must agree<\/strong> before any decisions can be made. This majority group is called a quorum<\/strong>.<\/p>\n
![\"\"](\"https:\/\/blog.techiescamp.com\/content\/images\/2025\/02\/image-6.png\")
<\/figure>\nA quorum is typically calculated as (N\/2 + 1) where N is the total number of nodes in your system. This ensures that you always have a majority.<\/p>\n
For example:<\/p>\n
- \n
- In a 5-node system, you need 3 nodes to form a quorum (5\/2 + 1 = 3)<\/li>\n
- In a 7-node system, you need 4 nodes (7\/2 + 1 = 4)<\/li>\n<\/ul>\n
Thats why clusters are usually deployed with an odd number of servers<\/strong>, starting from 3<\/strong>, to ensure that a majority (quorum) can always be formed<\/strong>.<\/p>\n
Real World Scenario (etcd)<\/h2>\n
According to the official etcd documentation, split-brain scenarios are effectively avoided in etcd.<\/strong><\/p>\n
![\"\"](\"https:\/\/blog.techiescamp.com\/content\/images\/2025\/02\/image-7.png\")
<\/figure>\nHere’s how:<\/p>\n
- \n
- When a network partition occurs, the etcd cluster is divided into two segments: a majority and a minority.<\/li>\n
- The majority<\/strong> segment continues to function as the available cluster, while the minority<\/strong> segment becomes unavailable.<\/li>\n
- If the leader resides within the majority segment, the system treats the failure as a minority follower failure. The majority segment remains operational<\/strong> with no impact on consistency.<\/li>\n
- If the leader is part of the minority segment, it recognizes that it is separated due to the loss of communication with a majority of the nodes in the cluster and it steps down from its leadership<\/strong> role.<\/li>\n
- The majority segment then elects a new leader, ensuring continuous availability<\/strong> and consistency.<\/li>\n
- Once the network partition is resolved, the minority segment automatically identifies the new leader from the majority segment and synchronizes its state accordingly.<\/li>\n<\/ol>\n
This design ensures that etcd maintains strong consistency and prevents split-brain scenarios, even in the face of network issues.<\/p>\n
You might ask, how does etcd determine if a node is in the majority or minority<\/strong> during a network partition?<\/p>\n
Each etcd node knows the total number of nodes<\/strong> in the cluster as part of its configuration. This is stored in the etcd membership list<\/strong>, which tracks:<\/p>\n
- \n
- Total cluster size (N)<\/li>\n
- Node IDs and addresses<\/li>\n
- Current leader (if elected)<\/li>\n<\/ul>\n
Nodes regularly send messages (heartbeats) to confirm which nodes are still reachable. After detecting a partition, each node checks how many other nodes it can still talk to<\/strong>.<\/p>\n
It compares this number with the quorum requirement (
N\/2 + 1<\/code>).<\/p>\nIf a node can still communicate with at least quorum (N\/2 + 1) nodes, it knows it is in the majority.<\/p>\n
If it cannot reach quorum<\/strong>, it realizes it is in the minority<\/strong> and stops making decisions.<\/p>\n
Conclusion<\/h2>\n
- If the leader resides within the majority segment, the system treats the failure as a minority follower failure. The majority segment remains operational<\/strong> with no impact on consistency.<\/li>\n
- Multi-Primary Node<\/strong>: In this setup, multiple nodes can accept writes simultaneously. When you write to any primary node, that node coordinates with others to ensure the write is properly replicated.<\/li>\n<\/ol>\n
- Single-Primary Node<\/strong>: All writes go to the primary node, which then replicates those changes to secondary nodes.<\/li>\n