Regional placement and survival

Regional placement decides where a shard’s copies live and where its requests run. Put its voting replicas close together and they can agree quickly. Spread them across regions and a regional outage can leave enough voters to continue, but agreement may have to cross the network between regions.

The number of copies alone does not tell you either result. Three voters in one region can survive a machine failure and all disappear in one regional failure. A customer’s order can sit near the application while its purchase still waits for stock owned elsewhere.

This article gives the shop from Sharding and distributed SQL a separate regional layout. North, West and East are invented regions. Mina’s customer shard A has its leader in North; the shared product shard P has its leader in West. We first place the voters for one shard, then follow the work that purchase 103 needs. The placements and network times are illustrative models, not observations from a running cluster.

Regions and failure domains

A failure domain is a set of resources that one incident can remove together. A failed machine removes the replicas on that machine. A zone outage can remove several machines sharing power or connectivity. A region outage can remove several zones. Placing copies on different machines helps only against failures that leave those machines independent.

These names describe the failures you intend to tolerate. Check the actual dependencies behind them: separate zone labels do not make a shared network or power supply independent. The same applies to the application. Surviving database voters are little use to a request if its only application server has disappeared.

Keep three locations distinct. The request origin is where the application sends the request. The owner’s leader handles the authoritative work for a shard in this model. Its voting replicas participate in agreement on that shard’s history. Moving the request origin does not move either the leader or its voters.

Counting the surviving voters

The parent article established a fixed group of three Raft voters whose majority is two. Here we ask which failures leave that majority. Placing all three in separate North zones differs from placing two in North and one in West: a zone failure and a regional failure remove different sets of voters.

Three voters in separate North zones leave two after one zone fails, but none after North fails. Two in North and one in West leave two after West fails and only one after North fails. The surviving West copy still holds data; it cannot make itself a majority by treating the missing voters as removed.

With one voter per region, any one region’s loss leaves two of the configured three. With five voters placed two in North, two in West and one in East, the majority is three. Losing North or West leaves three; losing East leaves four. All three single-region failures pass the count.

For a fixed group of N voters, the majority is floor(N / 2) + 1. If an incident removes F of them, the first test is whether N − F still reaches that majority. Count voters per shard, not machines across the whole cluster. Five healthy machines holding other shards cannot complete P’s majority.

The survivors must also communicate. Isolating North in the one-per-region layout leaves West and East able to form a majority together, while North alone cannot. If West and East cannot communicate either, there is no majority anywhere. An election, working storage and enough capacity remain necessary even when the count passes; the majority count establishes a possible path to progress, not a deadline for recovery.

Retaining history and accepting new writes are different promises. A group without a quorum may retain every earlier committed change and still stop new writes. A non-voting replica can add a copy, and may support qualified reads, but it does not add a vote to this calculation. A backup is another way to retain recoverable data; it does not answer a live agreement request.

What a regional write waits for

Spreading voters changes the healthy path too. Give P one voter per region, with West already its leader. To form a majority of two, West needs its own durable record and one other voter’s acknowledgement. That other voter must be remote.

Use the separate catalogue change from the parent: P7’s price rises from 20.00 to 22.00. Stock stays at its post-purchase value 0, and order 103 keeps its agreed line prices and total 29.00. Assume the command has reached West, with a West–North network round trip of 40 ms and a West–East round trip of 90 ms. North’s availability determines which remote acknowledgement can complete the majority.

In the healthy case, North’s acknowledgement completes West’s majority. The agreement path includes at least the assumed 40 ms network round trip. East need not answer first. With North unavailable, West needs East, so that network contribution becomes at least 90 ms. The group survives North’s loss, and the route that lets it survive is slower.

These are network contributions to one established leader’s agreement round. They are not predicted SQL commit latencies. Storage must make the required records durable; the leader must establish commitment and apply the command before replying under this model. Client routing, queues, execution and any transaction protocol add work. After a leader failure, detection and election add a different wait. The protocol’s event order alone gives no elapsed time.

A five-voter group placed two–two–one illustrates the same trade. A leader in a two-voter region can count both local voters and still needs a third vote from elsewhere. A larger group changes the required set of acknowledgements; it does not make distant agreement disappear. If the nearest useful voter is slow, a faster alternative may complete the majority instead.

The model follows Raft with fixed membership, an established leader and a new command from that leader’s current term. It inherits the parent’s election and log rules, and assumes non-malicious failures and storage that preserves acknowledged durable state. Merely counting stored copies of an older command is not a substitute for those rules.

A local order can need remote stock

Now return to before purchase 103. The North application handles Mina’s request. A must create her pending order and its two lines. P must guard the stock changes: P7 from 1 to 0 and P3 from 12 to 10. Both product rows belong to P in this layout, so there are two data owners.

Give both A and P one voter in every region, keeping A’s leader in North and P’s in West. Their leaders are where the purchase’s authoritative order and stock work runs; their other voters preserve each owner’s history.

The application reaches A locally, but A’s local order work is only part of the purchase. P’s stock authority is in West. Coordination between A and P crosses a regional boundary; agreement among A’s copies and among P’s copies can cross regional boundaries as well. The first is work between different owners. The second keeps copies of one owner on one history.

Those waits can overlap or require several rounds, depending on the database’s transaction protocol. Counting the dependencies does not produce a commit time: a purchase cannot safely claim success while a required owner is unable to complete its work or learn the joint outcome.

A and P agreeing separately does not establish one purchase outcome. Both need to participate in a recoverable atomic decision. Transactions across partitions follows that decision and its recovery. Here its consequence is placement: include every required participant when deciding which failures and network trips the purchase depends on.

Global constraints have a location

The shop’s order_id identifies one order globally. A unique check beside Mina’s rows can reject a second 103 on A. An independent check beside Tariq’s rows can reject a second 103 on B. Without a mechanism joining those checks, each can accept its first 103.

One variation adds a separate owner G to govern the globally unique number and its association with the order. G’s leader is in East, and G has its own three-region voting group. Its work must belong to the complete purchase protocol. Creating an unrelated reservation in East and then hoping the order write succeeds is not enough.

The extra owner is one possible design, not a claim about how every distributed database implements uniqueness. An engine might maintain a distributed unique index, use a specialised regional check, or reject a constraint whose scope it cannot enforce. A global index also has keys and replica placement, so inspect its work alongside the base rows. A duplicate-resistant ID generator reduces accidental collisions; by itself it does not reject an explicit duplicate supplied by another writer.

Changing the key to (region, order_id) changes the rule. It distinguishes (North, 103) from (West, 103), allowing the number 103 twice. That is regional uniqueness, and references must carry enough information to identify the intended order. It does not preserve the shop’s original ID-only identity.

References add work too. A new order line names P7, whose product identity belongs to P; a maintained local copy may or may not be sufficient for a particular engine’s reference checks. Include foreign-key enforcement and maintained secondary indexes before calling a write local. Copying catalogue details near A may save a read trip under a freshness contract. It does not allocate A an independent last blue mug. A different inventory-allocation protocol would have to preserve the total allowance across regions.

Reading near the application

A nearby copy can shorten a read without shortening the path that agrees a write. The read first needs a contract: must it reflect current authority, include a particular earlier write, or show an explicitly permitted past state?

Return to the independent price change. P’s West leader has replied successfully after West and East made the command durable, but North’s nearby copy has not applied it and still reads 20.00. That local answer must be judged against the particular read’s freshness requirement.

A read claiming current state cannot infer freshness from being close to the application. It needs the engine’s authority and freshness evidence, which may require remote work. Some protocols let a qualified replica serve it locally using previously established evidence; current reads do not universally require a new network round trip for every request.

A read required to include the acknowledged price change must use a target and snapshot qualified for that history. Waiting until a local copy has received a record is insufficient if it has not applied the change. Replicas, lag and read routing develops that distinction and the routing choices when a target is behind.

A historical catalogue view may permit 20.00, provided the engine can safely serve the requested past history. Permission to show an older price does not grant permission to sell against older stock. The purchase still checks and changes stock at its authority. A report joining regional owners also needs a compatible observation across them; Time, snapshots and distributed reads develops the common-snapshot machinery.

Choosing and checking a placement

Start with a request and a failure promise. For Mina’s purchase, “continue after any one region is lost” requires more than P retaining a majority. A, P and any required G must each retain a communicating majority. The surviving application must reach them, and the remaining machines must have capacity for their work.

State the latency target separately for healthy and degraded operation, and measure both. A majority count establishes structural survival under the stated failure model. It does not establish how quickly an election completes, how long transactions wait or how much throughput survives. For example, the one-per-region layout survives North’s loss while changing the assumed network contribution to P’s agreement from 40 to 90 ms.

Place copies for the failure you must survive, then place leaders and request work to make the remaining coordination acceptable. Check the actual constraints and replica assignments after configuration. CockroachDB table locality and region survival takes that reasoning into native settings and verification. Rebalancing and cluster change explains how to reach a new layout safely while copies catch up and membership changes.