Failover and fencing

A missing heartbeat does not stop a writer

A heartbeat is a repeated message used to check that another part is reachable. When it stops arriving, the primary might have failed. The link might also have failed while the primary continues serving callers on its side.

Suppose an application pool already has a connection to the primary. The monitor and replica A lose contact with that primary, but the application’s path still works. Moving the service address to A changes where new callers go; it need not close that established connection.

If A is promoted while the old connection remains usable, new callers reach A while the existing caller still reaches the primary. Both databases can commit work. This is split brain: two competing write histories for what the application treats as one database. Each copy can enforce its own keys and checks while the service as a whole has no single answer about accepted work.

Promoting A is not a way to discover whether the primary stopped. It creates the second writer if the first is still active. Waiting longer for a heartbeat can reduce false alarms, but the elapsed wait does not enforce exclusion.

Fence the former primary

Fencing prevents the former writer from continuing authoritative work. The replacement must not accept writes until the authority protocol establishes that exclusion. Sending a shutdown request is only an attempt; the handover needs evidence that the required stop or isolation has taken effect.

An independent power-control path can confirm that the old machine is off, with restart controls preventing it returning as a primary. A controlled switchover can instead confirm a clean shutdown and keep every supervisor from starting it again as a writer. When the machine cannot be trusted to respond, use a tested fencing mechanism that does not depend on that machine cooperating.

A network fence can work only if it closes the complete set of relevant paths, including already established connections and local workers. Moving one address is useful routing work after the transfer, but it leaves those other routes to the old primary open.

Leases and fencing tokens

A leader lease grants authority for a bounded interval, renewed through a coordination service. The old primary must stop committing before a successor can use that authority. A paused or overloaded process may not run its demotion code in time. Lease timing, clock assumptions and an enforced stop therefore belong to one tested protocol. Patroni’s watchdog design illustrates why a separate reset mechanism may be needed when ordinary demotion fails.

A fencing token carries a generation of authority to the resource receiving an operation. Suppose a hypothetical receiver durably remembers generation 8 for one resource. After accepting 8, it rejects a delayed request carrying 7, with the comparison and operation enforced together. Merely adding “8” to a request does nothing if the receiver ignores it. A token checked by one file service also does not stop an old database from committing to its own disk.

Agreement about the leader, enforcement at each affected resource and preservation of committed history must fit together. Sharding and distributed SQL develops replica agreement; Outboxes and database-backed jobs applies receiver enforcement to workers.

Choose the history to promote

Return to C, the acknowledged price edit. A still reads 20.00, but its durable log contains U and C. Once A replays that complete usable chain, a new snapshot can include 22.00. B also reads 20.00, but lacks the records needed to recover the edit. Equal query results hide different recovery possibilities.

The promotion decision must distinguish a candidate that can recover the confirmed edit from one that merely returns the same old price. It also needs an explicit boundary between the original confirmed history and the history the replacement will make writable.

A can retain this particular commit. That alone does not prove it contains every acknowledged commit from a real primary. Compare candidate progress within compatible histories, establish that the required records survive and can be recovered, and relate them to the actual acknowledgement policy. A larger number on a different branch is not evidence that it contains C.

State the loss boundary

If the price transaction waited for C to be durable on A, and A’s promised storage survives, A has the required copy. If it waited only for the primary’s local flush, no particular replica was promised to have C when success returned. Synchronous replication must name the required copies and progress, and promotion must respect that choice.

Suppose A is now unavailable too. B’s reachability does not inherit A’s durability guarantee. Preserving the confirmed price edit requires obtaining its missing history from a usable surviving source, waiting for a qualifying copy, or keeping writes unavailable. An emergency promotion of B without C explicitly accepts omission of that edit.

In this small schedule the loss is known: C is absent. In an incident, incomplete evidence may leave only a possible loss boundary. Record that uncertainty rather than turning the last lag sample into a promise. A recent small backlog can become stale while the primary continues writing. A majority that elects a controller also need not hold the database’s latest commits. Safe automatic promotion needs a protocol linking candidate eligibility to the commit promise.

Promotion is part of a handover

The checks can be prepared in parallel, but the replacement must pass the authority and history gates before serving writes. The concrete transfer has this order:

1. Exclude the former writer. Confirm the fence or completed demotion, including restart exclusion. Stop competing promotion attempts through the same authority protocol.
2. Establish usable history. Identify the candidate, its branch and recoverable progress. Obtain and apply the required records, or declare the loss being accepted.
3. Promote under the authority protocol. Verify that this exact server has the intended role and recovered facts before admitting ordinary callers.
4. Move callers and workers. Replace stale pooled connections, revalidate their target and account for requests already in flight. Enable external dispatch only with its outstanding work understood.

A promotion command changes a database role. It does not drain application pools, recover a missing commit or decide which external work already happened. A request that lost its commit reply remains uncertain across the transfer. Recover its outcome under its original identity using Retries and uncertain commit outcomes; do not label it aborted merely because its connection failed.

Check business facts alongside role and history evidence. This query checks P7’s current catalogue price separately from order 103’s agreement:

SELECT p.price AS catalogue_price, o.status,
       (SELECT sum(quantity * unit_price)
        FROM order_items WHERE order_id = o.order_id) AS line_total
FROM products p CROSS JOIN orders o
WHERE p.product_id = 'P7' AND o.order_id = 103;
-- A after C: 22.00 | pending | 29.00
-- B without C: 20.00 | pending | 29.00

These results were checked on constructed states in PGlite 0.5.8, PostgreSQL 18.3 compiled to WebAssembly. They check the query and example arithmetic. They do not demonstrate replication, power fencing or a live failover. A production readiness check also needs the actual server identity, authority, history and replication evidence.

Account for missing and divergent work

Promoting B without C chooses a new branch from an earlier prefix of H. Once new commits arrive on B, restarting the old primary cannot join the two histories by continuing log replay.

The common prefix remains in both copies. C belongs only to the former history; subsequent authoritative work belongs to the chosen new branch. Any extra work the old primary accepted before it was fenced must be accounted for too. Keep the former data and logs, plus independent request and external-effect evidence, before a rewind or fresh copy destroys that history.

Separate confirmed operations omitted by the cut, requests with uncertain outcomes and work already present on the new authority. Determine each operation’s identity, meaning, dependencies and accepted disposition. An omitted confirmed price edit is evidence to resolve, not permission to overwrite whatever catalogue decision the new authority has since made.

Reconcile accepted business operations against current constraints and record their outcome under the original identity. Blindly replaying SQL can repeat a stock decrement or apply a change to different data. Physical recovery logs are not a business-operation merge format. Backup, restore and reconciliation develops this accounting, including payments already accepted by a provider whose history the database transfer does not rewind.

Return the former primary as a replica

A machine coming back proves that it can run again. Keep it fenced from authoritative service while preserving evidence and making its data follow the chosen writer. A supported rewind may reuse unchanged data; otherwise create a fresh copy. Neither method recovers omitted business operations automatically.

Configure the returning server as a replica before starting it for normal use. Verify its role, the history it follows and catch-up, then verify routing and the protection required by the service’s commit policy. A reachable replica is not yet evidence that redundancy has been restored. PostgreSQL physical replicas and promotion owns the native promotion, rewind prerequisites and return procedure.

Practise the whole transfer

A planned switchover can stop admission, drain in-flight writes, establish the final source boundary and wait for the replacement to copy it before transferring authority. Failover must make decisions with missing evidence. Both need writer exclusion and a verified destination.

Rehearse a broken link while the old primary remains reachable to a caller, an unavailable required replica and a returning former primary. A drill that only stops the server cannot expose a still-connected caller writing to it. Include old pools, workers and uncertain replies, then restore the intended redundancy.

Keep evidence of which commits were retained or omitted, whether the former writer was actually excluded, how callers recovered, and how long useful service and full protection took to return. Those observations let you judge the next transfer against the service’s recovery objectives. A fast promotion is one event inside that recovery.