Database failover is not complete when a standby becomes primary. It is complete when clients discover the new writer, stale connections are discarded, writes are accepted at the expected durability level, and the application passes an end-to-end check. That broader recovery path is what a high-availability test must verify.
This guide explains how database failover and high availability work, the traps (split-brain, quorum, lost writes), and how to verify your setup actually recovers.
What High Availability and Failover Mean
High availability (HA) means a system is designed to keep running despite the failure of individual components — typically by having redundancy so there's no single point of failure. For databases, that usually means running a primary plus one or more standbys (replicas) that can take over.
Failover is the act of promoting a standby to become the new primary when the old one fails. After failover, the promoted standby accepts writes and the application reconnects to it. The goal is to minimize two things:
- Downtime — how long the database is unavailable during the switch (your recovery time objective, or RTO).
- Data loss — how many recent writes didn't make it to the standby before the primary died (your recovery point, tied directly to replication lag).
HA reduces both. It does not eliminate them — a failover is a disruption, however brief, and asynchronous replication means some data loss is possible. Anyone who promises zero of both is overselling.
Automatic vs. Manual Failover
There are two ways failover happens:
- Manual failover. A human detects the failure and promotes a standby. Slower (minutes, and only as fast as your on-call response), but safe — a person confirms the primary is really dead before promoting, avoiding false alarms.
- Automatic failover. A monitoring/orchestration layer (Patroni, Orchestrator, a managed service like RDS Multi-AZ, etc.) detects the failure and promotes a standby with no human in the loop. Fast (seconds to a couple of minutes), but only as good as its failure detection — and that's where the danger lies.
The hard problem in automatic failover is deciding whether the primary is truly down. A network blip between the orchestrator and the primary looks identical to a dead primary. Promote too eagerly and you risk split-brain; promote too cautiously and you extend the outage. This tension is the heart of HA design.
The Big Traps: Split-Brain, Quorum, and Lost Writes
Three failure modes turn a "highly available" setup into an incident:
Split-brain. If the old primary isn't really dead — just temporarily unreachable — and a standby gets promoted anyway, you now have two primaries both accepting writes. They diverge, and reconciling the conflicting data afterward is painful or impossible. Split-brain is the nightmare scenario of naive automatic failover.
Quorum and fencing. The defense against split-brain is a quorum of voting members plus fencing that prevents the former primary from accepting writes. Odd voter counts commonly make majority decisions possible across a single partition, but adding replicas does not automatically add safe voters; topology and failure domains must be designed for the database's actual consensus or orchestration system.
Lost writes. With asynchronous replication, writes committed on the primary but not yet replicated are lost when you promote a standby. The bigger your replication lag at the moment of failure, the bigger the data-loss window — which is why monitoring lag is a core part of HA, not a separate concern.
A fourth, quieter trap: the application doesn't reconnect. The database fails over perfectly, but connection pools cling to dead connections or DNS caches point at the old host, so the app stays down even though the database recovered. Failover isn't done until clients are talking to the new primary.
Test and Monitor the Complete Failover Path
HA that's never tested is just a hope. What to monitor and verify:
- Replication health and lag on every standby, so you always know a promotable, reasonably-current replica exists. A standby that silently stopped replicating is a failover that will lose data or fail outright.
- Cluster and quorum state. Watch that the expected number of nodes are healthy and that the cluster has a quorum — a degraded cluster may not be able to fail over at all.
- Failover events themselves. Alert when a promotion happens, both so humans know and so you can review whether it was correct.
- End-to-end recovery, from the outside. The real question isn't "did the database promote a standby" — it's "is the application serving correct responses again." Only outside-in monitoring answers that.
- Practice failovers regularly (game days / chaos drills). A failover you've rehearsed is routine; one you've never tested is a coin flip during your worst hour.
Failover Drill Checklist
- Record the primary, replica replay position, replication mode, and expected RTO/RPO.
- Start an external canary that writes a unique value and reads it through the same route used by production.
- Trigger a documented planned failover; do not test by improvising a destructive failure.
- Measure detection, promotion, endpoint/DNS update, client reconnection, and first successful write separately.
- Verify the old primary is fenced and cannot accept writes.
- Confirm the promoted node contains the last acknowledged canary write, then rebuild redundancy.
- Review connection errors, lost writes, duplicate operations, and alert timing before declaring success.
This keeps the failover article's intent distinct from replication-lag diagnosis: lag measures replica currency; the drill proves the recovery control plane and clients work.
How Webalert Helps
Internal cluster tooling knows the database's view of failover; Webalert provides the independent, outside-in view that tells you whether users actually recovered:
- End-to-end recovery confirmation. When a failover fires, Webalert verifies your real endpoints are serving correct responses again — catching the case where the database recovered but the app never reconnected.
- Independent outage detection that doesn't depend on the same infrastructure that's failing, so a cluster-wide problem can't blind your monitoring at the same time.
- Downtime measurement for the failover window, giving you real data on your availability and SLA instead of assumptions.
- Fast alerting so even a "few seconds" automatic failover that goes wrong reaches a human immediately.
Webalert won't run your failover, but it's the impartial witness that confirms your HA actually delivered availability — to users, not just on paper.
Primary Documentation
- PostgreSQL warm standby and failover — promotion, archive recovery, and failover behavior.
- MySQL Group Replication monitoring — membership and consensus-related state.
- MongoDB replica-set elections — election triggers, voting, and write availability.
- Amazon RDS Multi-AZ failover — managed failover events and client DNS implications.
Summary
High availability means designing a database so one failure doesn't cause an outage, and failover is the mechanism — promoting a standby when the primary dies — that delivers it. Failover aims to minimize downtime (RTO) and data loss (RPO), but eliminates neither: it's a disruption, and asynchronous replication means recent writes can be lost. Automatic failover is fast but risks split-brain if it misjudges a network blip for a dead primary; quorum, fencing, and odd node counts are the defenses.
The traps that turn HA into an incident are split-brain (two primaries diverging), lost writes from replication lag, and applications that never reconnect to the new primary. Monitor replication health on every standby, cluster quorum state, and failover events — but above all, verify end-to-end from the outside that users actually recovered, and rehearse failovers before you need them for real.