Failover is switching work from a component that has failed to a redundant one that is standing by, so the service keeps running. Database failover is the database version of that: when the primary database server fails, a standby replica is promoted to become the new primary, and applications reconnect to it and keep reading and writing.
The promotion is only half the job. A database failover is complete when clients have found the new writer, stale connections are gone, writes are accepted at the durability level you expect, and the application passes an end-to-end check. This guide covers the concepts, the architectures, the traps, and how to test and monitor the whole path.
What Is Failover?
Failover is an automatic or manual switch to a backup system when the primary one stops working: a load balancer moving traffic away from a dead web server, DNS pointing a hostname at a secondary region, a database standby taking over writes. Every failover setup needs three parts: redundancy (a second copy of what can fail), detection (a health check, cluster manager or person that notices the failure) and switchover (a way to move traffic to the backup and make clients use it).
Returning work to the original component once it is repaired is called failback. It is a separate, planned operation and deserves the same care.
What Is Database Failover?
Database failover is harder than failing over a stateless web server, because a database holds state that must stay consistent. You usually cannot let two database servers accept writes independently without risking conflicting data. So most databases use a primary that accepts writes and one or more standbys (replicas) that receive a copy of every change. When the primary fails, database failover means:
- confirming the primary is really unavailable,
- choosing the most up-to-date healthy standby,
- promoting it so it accepts writes,
- stopping the old primary from accepting writes if it comes back,
- moving clients to the new primary through a virtual IP, DNS change, proxy or driver-level discovery.
Each of these steps can fail independently, which is why a "successful" promotion can still leave users looking at errors.
High Availability vs Failover
The two terms are related but not interchangeable.
- High availability (HA) is a goal and a design property: the system keeps serving users despite the failure of individual components. You get it through redundancy, removing single points of failure, health checking and recovery automation. It is usually expressed as an availability target, such as 99.9% or 99.99% over a period (see uptime SLAs explained).
- Failover is one mechanism that delivers high availability. It is the act of switching from a failed component to a standby.
High availability is what you want, and failover is one of the ways you get it. An HA system also needs monitoring, capacity headroom, safe deploys and tested recovery procedures. And having failover does not make a system highly available: if failover takes twenty minutes, loses data or leaves clients connected to a dead host, availability still suffers.
A database with automatic failover and high availability therefore means a cluster that detects primary failure on its own, promotes a standby without waiting for a human, and is designed so that the promotion is safe and clients follow it.
Failover Architectures
"Failover architecture" covers a few independent choices. You pick one option from each, and the combination decides how much downtime and data loss you accept.
Active-passive and active-active
Active-passive (also called primary-standby) is the most common database setup. One node handles writes. Standbys replicate from it and may serve reads, but they do not accept writes until one is promoted. It is simpler to reason about because there is only ever one writer. The cost is that failover is a real event, with a short window in which no node accepts writes.
Active-active (multi-primary) lets more than one node accept writes at once, so losing one node does not stop writes. The cost is conflict handling: two nodes can change the same row concurrently, so the database or application must detect and resolve conflicts, or partition data so each node owns different rows. Active-active removes the promotion step but adds complexity elsewhere. For most applications, active-passive with a well-tested failover is the sensible default.
Manual vs automatic failover
Manual failover means a human decides the primary is gone and runs the promotion. It is slower, since recovery waits for someone to be paged and check the situation, but safer against false alarms.
Automatic failover means software decides and acts: Patroni for PostgreSQL, Orchestrator for MySQL, MongoDB's built-in replica set elections, or managed services such as Amazon RDS Multi-AZ. PostgreSQL itself does not detect primary failure; its documentation says it "does not provide the system software required to identify a failure on the primary", which is why tools like Patroni exist.
The hard problem in automatic failover is deciding whether the primary is really down. A network partition between the orchestrator and the primary looks the same as a dead primary. Promote too eagerly and you risk two writers; too cautiously and you extend the outage.
Some published figures, for scale:
- Amazon's RDS documentation says Multi-AZ DB instance failover times "are typically 60–120 seconds", and that large transactions or a lengthy recovery can increase them (AWS docs).
- MongoDB's documentation says the median time to elect a new primary "should not typically exceed 12 seconds" with default settings, and that the replica set cannot process writes until the election completes (MongoDB docs).
Those figures cover the database side only. Your users' downtime also includes the time for clients to notice and reconnect, which you have to measure yourself.
Synchronous vs asynchronous replication: RPO and RTO
How changes reach the standby determines how much data a failover can lose. Two objectives frame this (see RTO vs RPO explained):
- RPO (recovery point objective): how much recent data you can afford to lose.
- RTO (recovery time objective): how long you can afford to be unavailable.
Asynchronous replication confirms the commit as soon as the primary has it; the standby receives the change shortly afterwards. Writes are fast and do not depend on the standby, but if the primary crashes, transactions not yet shipped are gone. PostgreSQL's replication documentation says it directly: with the default asynchronous mode, "some transactions that were committed may not have been replicated to the standby server, causing data loss." Your effective RPO is your replication lag at the moment of failure.
Synchronous replication waits for a standby to confirm it has the change before telling the client the commit succeeded. In PostgreSQL's synchronous mode, each commit waits until it is written to the write-ahead log on disk on both primary and standby, and the docs state that data can then only be lost if both crash at the same time. The costs are higher commit latency, since every write waits on a network round trip, and a dependency on standby health: if the synchronous standby is unavailable, writes can stall unless another is configured.
RTO is driven more by detection and client reconnection than by replication mode. A slow health check or a client that caches DNS forever will dominate recovery time either way.
A common middle ground is a synchronous standby in a nearby zone plus asynchronous replicas further away for disaster recovery.
The Traps: Split-Brain, Quorum, Fencing and Lost Writes
Split-brain. The old primary was not dead, only unreachable, and a standby was promoted anyway. Now two primaries accept writes, the data diverges, and reconciling it afterwards is painful or impossible.
Quorum. The defence against split-brain is to require a majority of voting members to agree before promoting. With three voters in three failure domains, any single failure leaves a majority on one side, and only that side can elect a primary. Adding replicas does not automatically add safe voters: two voters in one zone and one in another lose quorum if the first zone fails.
Fencing. Quorum stops a minority electing a new primary, but not an old primary that is still running from accepting writes. Fencing does that: revoking its storage, cutting its network, shutting it down, or having it step down when it loses the majority. PostgreSQL's docs refer to this as STONITH ("Shoot The Other Node In The Head") and note that without it both systems may think they are primary, leading to data loss.
Lost writes. Writes committed on the old primary but not yet replicated are lost on promotion. If the old primary later rejoins, those writes must be discarded or reconciled, not silently merged.
Clients that never reconnect. The database fails over cleanly, but connection pools keep handing out dead connections, or a DNS cache keeps resolving the old address. AWS notes that RDS failover changes the DNS record, so existing connections must be re-established, and recommends a JVM DNS TTL of no more than 60 seconds because some Java configurations cache lookups until restart. Check pool settings, connection validation and retry logic (see connection pool exhaustion and retry storms).
Replica currency. A standby that silently stopped replicating hours ago still answers queries. Promote it and you lose hours of data. Monitor replication on every promotion candidate continuously.
How to Test High Availability and Failover
Untested high availability is an assumption. Testing should prove the whole recovery path, not just that the database can promote a node.
HA testing checklist
Before you trust a setup, confirm each of these:
- Every standby is replicating, and its lag is measured and alerted on.
- Voting members are spread across separate failure domains and the cluster has quorum.
- The old primary is fenced, or steps down on its own, when it loses quorum.
- Applications connect through a stable endpoint (virtual IP, DNS name, proxy or a cluster-aware driver), not a hard-coded host.
- Connection pools discard broken connections, and DNS caching is short enough for your RTO.
- Write paths that may be retried after a failover are idempotent (see idempotency keys).
- Failover events page a human, even when the failover is automatic.
- RTO and RPO targets are written down, and there is a runbook for manual failover and failback.
Game-day steps
A game day is a scheduled, supervised failover exercise. Start with planned switchovers in staging, then production during low traffic. Chaos engineering covers the broader practice.
- Record the baseline. Note the current primary, each replica's replay position and lag, the replication mode, and the RTO and RPO you expect.
- Start an external canary. Run a check from outside your infrastructure that writes a unique value and reads it back through the same route production traffic uses.
- Trigger a documented failover. Use the supported switchover command or the provider's forced-failover option. Do not improvise destructive failures until planned ones are boring.
- Time each phase separately. Detection, promotion, endpoint or DNS update, client reconnection, and the first successful write.
- Verify fencing. Confirm the old primary cannot accept writes, including after it is restarted.
- Check for data loss. Confirm the new primary contains the last canary write that was acknowledged before the failure.
- Review the side effects. Look at connection errors, failed requests, duplicate operations from retries, and whether alerts fired at the right time to the right people.
- Restore redundancy. Rebuild or rejoin the old primary as a standby before you call it done.
- Escalate the scenario next time. Once planned switchovers pass, test harder cases: killing the primary process, partitioning the network, failing a whole zone, or failing over under peak-like load.
Write up each game day like an incident, with timings and follow-ups (the post-incident recovery checklist helps).
What to Monitor Before, During and After Failover
Before: is failover even possible?
- Replication state and lag on every standby. This is your live RPO. See the PostgreSQL, MySQL and MongoDB monitoring guides for the specific metrics.
- Cluster membership and quorum. A cluster that has already lost one voter may not be able to survive the next failure.
- Synchronous standby availability, if you use synchronous replication, since its loss can stall writes.
During: is it happening, and is it working?
- Failover and election events. Alert on every promotion, automatic or not. Managed services such as RDS publish these as events you can subscribe to.
- External uptime checks on the endpoints that depend on the database. These tell you when users started seeing errors and when they stopped. A health check endpoint that runs a real query against the database is more useful here than one that only returns 200.
- Application error and connection-error rates, to see whether clients are reconnecting or stuck.
After: did users actually recover?
- End-to-end checks pass again, including a check that validates the response body rather than only the status code (see response body validation).
- Response times. A new primary with a cold cache, or in a different zone from your app servers, shows up as a step change in response-time checks. Latency percentiles show this better than averages.
- Redundancy restored. Until a new standby is replicating, you are one failure away from a full outage.
- Measured downtime. Record the outage window from the external checks, not from the database's own logs, and compare it with your RTO.
Checking from more than one region helps you tell a real failover outage apart from a network problem on the monitoring side.
How Webalert Helps
Your cluster tooling knows the database's view of a failover. Webalert gives you the independent, outside-in view of whether users recovered:
- Uptime and content checks on the pages and API endpoints that depend on the database, confirming they serve correct responses again after a promotion. This catches the case where the database recovered but the app never reconnected.
- Response-time tracking that shows a slow new primary or a cross-zone latency jump after failover.
- Downtime measurement for the failover window, giving you real data for your RTO review and your availability reporting.
- Alerts by email, SMS, Slack and more, so a failover that goes wrong reaches a person quickly.
Webalert will not run your failover. It confirms the failover delivered availability to users, not just on paper.
FAQ
What is failover in a database?
Database failover is promoting a standby replica to become the new primary when the current primary fails, then moving applications to it so they can keep reading and writing.
What is the difference between high availability and failover?
High availability is the goal: keeping a service running despite component failures. Failover is one mechanism for reaching it, by switching to a standby when the primary fails.
What is automatic failover?
Automatic failover is when software detects that the primary has failed and promotes a standby without waiting for a human. It shortens recovery time but needs quorum and fencing to avoid promoting a standby while the old primary is still accepting writes.
Does database failover lose data?
It can. With asynchronous replication, transactions the primary committed but had not yet shipped to the standby are lost when the standby is promoted. Synchronous replication avoids this for acknowledged writes, at the cost of higher write latency.
Primary Documentation
- PostgreSQL failover: promotion, and the need to stop the old primary returning as a writer.
- PostgreSQL log-shipping and streaming replication: asynchronous and synchronous replication and their data-loss guarantees.
- MySQL Group Replication monitoring: membership and group state.
- MongoDB replica set elections: election triggers, voting and write availability.
- Amazon RDS Multi-AZ failover: failover reasons, timing and DNS behaviour.
Summary
Failover is switching to a standby when a primary fails. Database failover adds the harder parts: choosing a current standby, promoting it, fencing the old primary and getting clients to reconnect. High availability is the broader goal; failover is one way to reach it.
Active-passive is simpler, active-active avoids promotion at the cost of conflict handling, automatic failover needs quorum and fencing, and asynchronous replication trades possible data loss for faster writes. Test the full path with regular game days, monitor replication and quorum before failover, and use external uptime and response-time checks to confirm users actually recovered.