Rebalancing and cluster change

What rebalancing moves

A leader transfer changes which existing replica coordinates a group’s work. It can move request load without moving the stored data. The new leader must qualify under the engine’s authority protocol; directing clients at another process does not give that process authority.

A replica move changes where a copy of the same shard lives. Replacing R3 with R4 leaves P owning the same products. Its agreed history must survive the replacement. The move can change storage use and which machine or regional failures P survives.

A shard ownership transfer moves a portion between independently managed owners. Application routing, incoming references and concurrent writers may need a handover as well as a data copy. Distributing PostgreSQL work develops that transfer for application and PostgreSQL distribution choices. Here we keep P’s logical boundary fixed and change its replicas.

The distinction matters when adding a machine. Moving a leader can relieve a busy request path, provided a suitable replica already exists there. Moving stored bytes first adds work to the machines that have them. And neither move divides contention on the same last blue mug: that stock decision still belongs to one authoritative history.

Copying a live partition

A live copy needs a consistent starting state and a continuation from that state. Call the starting snapshot b. It represents P after an agreed prefix of history, with P7’s catalogue price still 20.00. Copying arbitrary files or rows while they change would not establish that boundary.

Reuse the separate catalogue change from the parent article. While R4 receives the base, the old group agrees a command U that changes P7’s price:

UPDATE products
SET price = 22.00
WHERE product_id = 'P7';

P7’s stock stays 0. Order 103 stays pending, with no payment and agreed order lines totalling 29.00. Only the catalogue price changes. Completing R4’s base copy still leaves it behind that change.

At the end of the base transfer, R4 has 20.00. The old group already has 22.00. The copy is internally consistent, but it is consistently from before U. R4 needs the compatible history after b, in the correct order, before it reaches the same state.

Receiving that continuation durably is another checkpoint. R4 can have U safely stored in its log while its applied state still says 20.00. It can recover the change after a restart, but has not yet made it available to reads. The distinction between durable and applied progress is the one developed in Replicas, lag and read routing. A count of copied bytes describes neither applied freshness nor voting authority.

Keeping the continuation available

The source must keep a way to bring the destination forward. A snapshot belongs to an identified history boundary; its replication metadata tells the engine where the following records attach. In Raft, that includes the last included log index and term, plus the configuration at that point. The letters b and U here are symbolic markers, not native log positions.

History cannot be retained without limit. If the required continuation has been discarded, the engine may need to send a newer snapshot and start that part of catch-up again. A stalled transfer can therefore spend bandwidth without ever becoming a current replica. Monitor progress towards the current history, rather than treating “copy started” as evidence that completion is approaching.

The old voters stay authoritative during this copy. R4 joins as a learner, a replica that receives history without participating in voting majorities. This avoids counting an empty or badly lagging destination as an available voter. Catching up prepares R4 for a membership change; it does not perform the change.

Changing voting membership

The old voting set is {R1, R2, R3}. The intended new set is {R1, R2, R4}. Each has a majority of two, but that does not make an uncoordinated switch safe. Individual replicas can learn a new list at different times.

Suppose a network split leaves R2 communicating only with R3, and R1 only with R4. If the first pair uses the old list while the second uses the new list, each can count itself as a majority.

R2 and R3 are two of the old three. R1 and R4 are two of the new three. The sets of actual voters do not overlap. Overlap between the full membership lists is therefore too weak: the transition must prevent these two majorities from acting independently.

Joint consensus

Joint consensus is the Raft paper’s solution. An intermediate configuration requires a majority of the old set and a majority of the new set for agreement and elections. It connects the two histories of authority before the old set is released.

The intermediate rule changes what either isolated pair may do. Each must satisfy both membership lists, even if it could make progress under one list alone.

R2 + R3 passes the old count and fails the new one. R1 + R4 does the reverse. Neither pair can act alone under joint rules. R1 + R2 passes both counts, because the same two replicas belong to both sets. Joint consensus requires two majorities, not two disjoint collections of acknowledgements, and not simply three of the combined four replicas.

The configuration entries themselves travel through the replicated log. In this version of Raft, a replica uses the latest configuration in its log as soon as it receives it. The joint entry is committed using both majorities. Only after that commitment may the leader propose the final new configuration, which commits under the new set’s rules. Once that final entry is committed, the old configuration is no longer required.

Use the database’s supported reconfiguration mechanism. Engines can use other safe protocols, including restricted changes one voter at a time; the joint model is an explanation, not a command sequence to apply to every Raft system. Deleting a missing voter’s files or changing a client’s routing list is not a replacement for the membership protocol. Primary/standby promotion and fencing have their own authority handover in Failover and fencing.

Preserving the survival margin

A safe membership protocol preserves one authoritative history. Availability through the move needs another check: which additional failure can the intermediate placement survive?

Suppose R3 is already unavailable. R1 and R2 can still form the old majority, but there is no remaining allowance for another old voter to disappear before the membership changes. A learner at R4 may hold useful data; it supplies no vote to that old majority. Stopping R2 because “three other copies exist” would leave only R1 able to vote.

Repeat the count at the joint checkpoint. R1 + R2 can satisfy both sets while R3 is unavailable. If R1 then fails, R2 + R4 supplies the new majority but only one old voter. Joint rules still need the old majority. After the completed new configuration, the same R2 + R4 set can pass. The timing of the extra failure changes the result.

Failure domains matter as well as replica names. If R4 is on a separate machine beside R1 in the same zone, a zone outage can remove both. The completed new set would then have only R2. If R4 replaces R3 in an independent zone, that particular shared failure is avoided. Regional placement and survival develops the placement tests; apply them to every intermediate configuration here.

State the survival requirement before the work starts: for example, retain a communicating quorum through one additional machine failure. Check it per affected shard, including groups sharing the machine being retired. A cluster-wide count of healthy machines cannot supply a missing vote for P. Passing the count also assumes working storage, communication, elections and enough resources; it does not promise an outage duration.

Capacity during rebalancing

During the move, the source serves ordinary requests while reading a base copy and sending later history. The destination writes that copy, stores incoming history and applies it. Temporary copies and retained history consume space until the old replica can be removed. The operation first increases work and storage use, even if its eventual purpose is to reduce both on one machine.

Budget the limiting resource at each location. A spare destination disk does not make the source’s network link spare. A lightly used processor does not make a saturated storage device faster. Copy traffic can also interfere with the small replication messages that keep existing voters responsive.

Catching up while writes continue

Reuse the earlier illustrative catch-up numbers: 600 MiB remains after the base copy, new history arrives at 10 MiB/s, and replay can apply 20 MiB/s. Only the difference, 10 MiB/s, reduces the old backlog. If foreground work reduces replay capacity, the same arrival rate consumes a larger share of it.

At 20 MiB/s replay, the ideal catch-up time is 600 ÷ (20 − 10) = 60 seconds. At 10 MiB/s the backlog never shrinks. At 8 MiB/s it grows, reaching 720 MiB after a minute. These are arithmetic results under fixed rates, not measured speeds or completion guarantees.

Replay rate alone is insufficient if the network delivers history more slowly. Use compatible units and measure the whole path: available history, delivery, durable receipt and application. If generation outruns any required stage, the destination falls behind there. A snapshot restart, bursts, compaction or foreground contention can invalidate a simple estimate.

Throttling base transfers or moving fewer replicas at once can protect foreground service. It can also prolong the period before redundancy is restored. Choose the rate and scope against both requirements: acceptable request latency and enough progress to complete safely. If the current budget cannot satisfy both, add capacity, reduce admitted work or change the maintenance window. Waiting is not a cure for a growing backlog.

Capacity after removing a node

The destination must fit after the move as well as during it. For a separate storage calculation, suppose the replica being moved occupies 120 GiB. Reserve space for expected growth, temporary work and the recovery work required by the declared failure promise.

D4 and D5 have 480 GiB free between them, but neither is an acceptable destination for this replica under the stated rules. D6 fits it while retaining the reserve. That establishes only this storage calculation. Its processor, network, memory and storage throughput must also support the final work and the required recovery load.

After removing a node, the surviving machines hold more data or serve more work. Recalculate the reduced cluster, including the busiest constrained destination and the next failure it must survive. Even replica counts can hide unequal sizes or hot keys. Connections, overload and capacity develops workload budgets and overload behaviour; movement spends part of that same budget.

Decommissioning a node

Decommissioning permanently removes a node from the cluster. Our R3 move concerns one replica group. Its machine can hold replicas of many groups, lead some of them, and accept application connections. All those responsibilities need a destination or an orderly end.

Draining moves or finishes active work so a process can stop. It does not by itself prove that stored replicas have left. For example, CockroachDB distinguishes draining connections and active roles from the heavier decommissioning operation that moves replica data. Other systems expose different procedures; inspect what each completion signal actually covers.

A request being accepted is the start of that procedure. A transfer at 100% is one part of it. The useful completion evidence ties the old node to every affected group and role:

If progress stalls, identify the missing condition. Does the destination’s applied position advance? Is the continuation still available? Is there an eligible store with room? Can the current group form the quorum needed to change membership? Repeating the removal request cannot create a destination or a missing majority.

Remove one planned unit at a time unless the complete concurrent operation has been budgeted and tested. Two independent maintenance requests can share a voter, a zone or a constrained destination. Each request looking safe alone does not make their overlap safe.

Operating and securing a CockroachDB cluster supplies native removal commands and health evidence. Upgrading a CockroachDB cluster handles binary rollout and cluster-version compatibility. For either procedure, the general test remains concrete: the replacement has the required history, the protocol has transferred authority, and the remaining placement can carry its promised work and failures.