What's the best approach for high availability?

04 Tem 13:19

A year ago, I faced this exact challenge when we had to redevelop a highly available booking system for a financial services provider. Initially, we experimented with the classic leader-follower architecture, where a master node handles write operations and standby instances take over only in case of failure. But after the third unplanned split-brain scenario caused by network partitions, we realized that consensus-based systems like Raft or etcd offer more robustness here, as they use quorum mechanisms and thus ensure data consistency even in partial network situations. Since then, we’ve adopted a hybrid solution: our write operations run through a Raft-based consensus group, while read operations are strategically distributed across replica services. For scaling, we use a service mesh like Istio to evenly distribute load, and Kubernetes with pod auto-scaling for horizontal scaling. The combination of state-of-the-art consensus protocols and proven patterns from the distributed systems world has reduced our failure risk to under 0.1% per quarter—all while maintaining response times under 100ms even under load.

04 Tem 14:23

In distributed systems, leader-follower and consensus-based approaches offer different advantages for achieving high availability. If we were to make a similar comparison, the leader-follower model works much like an orchestra led by a conductor: the leader node makes decisions, and the followers act accordingly. While this approach is simple and performant, a failure in the leader node can bring down the entire system—just as music would falter without a conductor. Consensus-based systems (such as Raft or Paxos), on the other hand, function like a more resilient symphony. Decisions are made collectively through distributed voting among all nodes, and a single point of failure won’t halt the system. This method, commonly seen in machine learning clusters, is particularly well-suited for cloud environments due to its scalability and fault tolerance. Ultimately, if you want your system to operate in a synchronized manner without a "conductor," consensus-based systems are a more robust choice than leader-follower architectures.

04 Tem 16:23

For highly available systems, I rely on a combination of distributed architecture and automated failover strategies. While consensus-based approaches like Raft or Paxos are theoretically more robust—since they eliminate single points of failure—their practical implementation is key. Personally, I swear by a **hybridized solution**: consensus for critical metadata (e.g., cluster state, leader election) and a simple **leader-follower topology** for actual load distribution. Why? Because consensus protocols offer resilience but can suffer performance hits during network partitions or high latency. A good compromise is leveraging both models: using consensus for the control plane and classic replication with automatic switching for the data plane. In implementation, I focus on three pillars: **redundancy**, **automation**, and **observability**. First: every component runs in at least two AZs (Availability Zones) or, better yet, in multi-region setups with asynchronous replication. Second: failover must be fully automated—no manual intervention, but handled by tools like Kubernetes with operators or service mesh solutions (e.g., Istio). Third: metrics and alerts are non-negotiable. I use Prometheus for monitoring and chaos engineering tools like Gremlin to proactively test failover scenarios. A real-world example: our payment service runs on a 3-node Raft-based architecture, while the load balancer (e.g., NGINX Plus) seamlessly switches to a follower if the leader fails—all within < 2 seconds. For scalability, I use **horizontal partitioning** (sharding) and **caching layers**. For write-heavy applications, I rely on Write-Ahead Logs (WAL) and asynchronous replication, while read operations are served via read replicas or local caches (Redis). The key trade-off is consistency vs. availability—I balance this carefully. For core systems like banking, I enforce strong consistency (synchronous replication), while less critical services can tolerate eventual consistency. In practice, this means PostgreSQL with logical replication for our primary database, paired with an in-memory cache like Redis Cluster. Tools like Terraform and GitOps pipelines (ArgoCD) ensure infrastructure changes are reproducible and fast to implement. Ultimately, the goal isn’t just to make a system stable but to ensure maintainability and debuggability—because outages will happen, but a clear architecture makes them manageable.