Upgrading a CockroachDB cluster

A CockroachDB major-version upgrade changes two things: the software running on each node, then the cluster’s internal compatibility version. The binaries can all be new while the cluster version is still old. That interval gives you a place to test and, on a supported path, return to the previous binaries. Beginning finalisation ends that option.

The difficult decision is therefore not merely whether the next process starts. It is whether the cluster can spare another node, whether the shop still works through its deployed clients, and whether the evidence justifies beginning an irreversible compatibility change.

This article follows a self-hosted v26.1.8 → v26.2.8 rehearsal. Inside CockroachDB supplies ranges and replicas; Operating and securing a CockroachDB cluster supplies secure identity, health and draining. Cloud and operator-managed deployments have their own upgrade controls. The versioned vendor procedure remains the authority for the deployment and release pair you actually run.

Binaries and cluster version

A binary version identifies the executable running on a node, including its patch release. The cluster version controls compatibility changes that need coordination across the cluster. During a supported major upgrade, old and new binaries can work together using the old cluster version.

Consider three nodes, A, B and C, initially running v26.1.8 with cluster version 26.1. Replacing A, then B, then C can leave all three on v26.2.8 while the cluster version remains deliberately held at 26.1. That is a different state from finalising the cluster at 26.2, even though the running binaries are the same.

The all-new, held state is useful precisely because it is unfinished. You can exercise the new software before enabling changes that older software cannot handle. It does not make the new binary behave exactly like the old one: a bug fix, a planner change or an ungated SQL change may already affect the application. Read the release notes to decide which behaviours can be accepted during the hold and which need finalisation.

Do not use one gateway’s build as a cluster inventory. SELECT version() describes the node serving that SQL connection. A connection entering A can perform useful work with replicas elsewhere while C still runs the older binary. Check all expected nodes and their identities.

SELECT version();
SHOW CLUSTER SETTING version;
SHOW CLUSTER SETTING cluster.auto_upgrade.enabled;
SHOW CLUSTER SETTING cluster.preserve_downgrade_option;

The two finalisation settings explain the policy as well as the observed version. Read the complete version value, including any upgrade-step suffix, and the migration state. An old major-version prefix does not establish that the upgrade remains deliberately held.

Prepare the upgrade and its return path

First establish the supported source and target, including any required intermediate release and patch requirements. This article’s adjacent 26.1 → 26.2 path does not establish permission to skip an arbitrary future release. Read the major release changes and the chosen patch notes, including backwards-incompatible behaviour, settings, deprecations and known limitations.

Make the shop’s current state a baseline before changing software. Keep the exact build inventory, cluster version, finalisation policy, schema and configuration with the acceptance results. A tested backup must cover what recovery would need, with a restore target compatible with that backup. A backup taken by a newer release is not a route to restoring on an older one.

Next check the cluster that will serve requests while a node is absent. Resolve unavailable and under-replicated ranges before beginning. Confirm the expected live membership, usable storage and resource margin. Three nodes in a list are insufficient if one affected range has already lost a voter. Rebalancing and cluster change develops the redundancy and capacity budget; stopping a node spends that same margin.

Choose finalisation policy before the first restart. Self-hosted automatic finalisation is enabled by default. With that policy, the last node’s return on the new binary can begin finalisation before you have completed application acceptance. For a deliberate testing interval, the current documentation recommends:

SET CLUSTER SETTING cluster.auto_upgrade.enabled = false;

Run cluster-setting changes as separate requests, with the authorised operator connection. Read the value back and retain the evidence. An existing cluster.preserve_downgrade_option = '26.1' also prevents automatic finalisation; it is the older control used in this rehearsal. Account for both controls when deciding how to begin finalisation later.

Verify licensing prerequisites as part of preparation. The v26.2 self-hosted procedure requires a valid licence for a major-version upgrade; some licence types also have telemetry requirements for finalisation. A local cluster accepting new binaries does not establish that it qualifies to finish. Discovering a missing prerequisite is a reason to stop at the held boundary.

Restart one node at a time

Remove the chosen node from new application traffic, let active work drain through the deployment’s supported shutdown procedure, then stop its process. A temporary upgrade preserves the node’s store and identity for its return. It is different from permanently decommissioning the node and evacuating its data.

Replace the executable or container image with the intended build and restart using the same store and the required configuration. The process manager must allow an orderly shutdown and must not start two processes against that store. Inspect the returning node itself, rather than accepting a successful query that a load balancer sent elsewhere.

A small query on A establishes SQL readiness:

SELECT 1;

Then verify rejoined membership, the expected build and fresh range health, using node status and the DB Console. Inspect resource pressure and the application’s completions before authorising B’s stop. The native command below shows node builds and range counts; use the same verified operator certificate and cluster settings as the operating procedure:

cockroach node status --ranges \
  --host=db-1.shop.example:26257 \
  --certs-dir=/run/shop-operator/certs

A fixed pause cannot supply this checkpoint. Waiting a minute does not repair an unavailable range, a failed rejoin or a client pool that never reconnects. If progress or service is unacceptable, leave the other nodes running and investigate. Repeat the complete checkpoint after each node, including the last.

Test while the old cluster version is held

Once every node runs v26.2.8, confirm that the cluster version remains 26.1 and the hold still exists. Now test the application’s actual contract, including its login, driver, prepared statements and connection pool. A privileged SQL shell cannot establish those client paths.

The shop gives the checks a concrete answer. After purchase 103, stock is P3 10, P7 0 and P9 4. Order 103 is pending, its agreed lines total 29.00 and it has no payment. In a separate catalogue transaction, raise P7’s current price to 22.00. Its order-line price remains 20.00.

UPDATE products
SET price = 22.00
WHERE product_id = 'P7';

SELECT sum(quantity * unit_price) AS agreed_total
FROM order_items
WHERE order_id = 103;
-- 29.00

Check a rejected operation as well as a successful one. With no blue mugs left, this guarded decrement must return no rows and leave stock at zero:

UPDATE products
SET stock = stock - 1
WHERE product_id = 'P7' AND stock >= 1
RETURNING product_id, stock;

Run the permitted runtime and support operations and their forbidden counterparts. For the access design in the operations article, support should still read only its selected tenant-scoped views, and still fail to read payments or change orders. Test fresh connections and the pool’s error paths. A disconnect during commit needs the existing retry and uncertain-outcome protocol; do not declare the transaction aborted merely because its gateway restarted.

Some release changes can only be tested after finalisation. In v26.2, checking the view owner’s underlying privileges and using the owner’s row policies are finalisation-dependent changes. The release notes identify them explicitly. A view that worked during the hold can therefore behave differently after finalisation. Put that view path on both sides of the acceptance plan; passing the first test does not predict the second.

Return to the previous binaries

Before finalisation has begun, the supported return repeats the rolling replacement with the previous binaries. Keep the old cluster version deliberately held, preserve each node’s store, and pass the same health and service checks between nodes. Confirm the documented return rules for this exact version pair before relying on them.

The price change makes the meaning visible. Returning A, B and C to v26.1.8 leaves P7 at 22.00. The database keeps the committed shop history; the executable replacement does not undo that transaction. It also does not reverse an application deployment or an independently committed schema change. CockroachDB schema jobs under load owns those application compatibility changes.

Rehearse this route on a copy before the live upgrade. Check every returned build, range health and application result. A return procedure that was written down but never exercised supplies less evidence than the successful forward rollout.

The native rehearsal started three v26.1.8 processes on one host. The rolling replacement produced mixed binaries, then three v26.2.8 binaries, while the cluster version stayed held at 26.1. Returning one node at a time reached v26.1.8 on all three again. The run committed the separate 22.00 catalogue price before its first restart; it remained 22.00 after the return.

Each completed checkpoint verified the expected per-node builds, live and available node status, zero unavailable and under-replicated range counts, and the shop projection through each node. Stock stayed 10/0/4, order 103 stayed pending with lines totalling 29.00 and no payment, and the guarded mug decrement affected zero rows. These are small SQL checks on one host, without production load, secure-client acceptance or independent failure domains.

Finalise the cluster version

Finalisation starts internal migrations that enable the new cluster compatibility. It is a separate decision after every node and the application have qualified. The critical boundary is when finalisation begins, not when the displayed cluster version finally changes.

For the older preserve control used here, first ensure automatic finalisation is disabled, then clear the preserve setting as its own request. After checking the decision and prerequisites, request the target version:

RESET CLUSTER SETTING cluster.preserve_downgrade_option;

Then send the version change separately:

SET CLUSTER SETTING version = '26.2';

Follow the migration jobs in the DB Console and retain their identifiers, statuses and errors. These are internal upgrade migrations, distinct from the shop’s application schema jobs. During migration, the reported version has not yet reached the final 26.2 value. It can include an intermediate upgrade-step suffix, as the native run below shows. An old major-version prefix does not reopen the binary-return route.

Completion needs successful migration outcomes and the intended active version, then the post-finalisation application tests. A terminal failed job is not success. The amount of internal work varies with the release and cluster; no elapsed wait substitutes for its result.

If the request fails a prerequisite, preserve its exact error and determine whether migrations began. Do not label an unchanged version as proof that returning binaries is safe. An uncertain boundary calls for diagnosis, not an experiment with older software against the stores.

The rehearsal then rolled forward again to three v26.2.8 binaries and requested finalisation. Its SQL client reached the test’s request timeout; the cluster continued working. The next version observation was 26.1-upgrading-to-26.2-step-016. Migration jobs included completed steps and a running step 018. This establishes that finalisation began, even though the final 26.2 value had not appeared. No older-binary return was attempted after that request.

The bounded run did not establish finalisation completion or the finalisation-dependent view checks. A client timeout is not a failed server migration, and this observation does not establish a stall. Completion at cluster version 26.2 remains the documented target, not an observed outcome of this rehearsal.

When the upgrade cannot continue

Do not force an older binary onto a store whose compatibility has advanced. Nor is a new-version backup a shortcut around restore compatibility. Recovery to the previous release needs an earlier compatible copy and a controlled transfer of write authority.

That copy can omit valid later work. If it predates the 22.00 catalogue change, restoring it brings back 20.00 until the intended change is reconciled. Valid orders, stock decisions and external payment effects would need the same attention. CockroachDB backup and recovery owns the restore, fencing and reconciliation procedure. Replacing binaries and restoring business history solve different problems.

Patch upgrades and future releases

A patch change within one major release uses rolling binary replacement and the same readiness, health and application checks. It does not add a separate major-version finalisation decision. Still read its patch notes and check its documented return path; a smaller version number change does not remove the need for acceptance.

For a future major upgrade, repeat the release-path, compatibility and recovery preparation. Settings, prerequisites and eligible version paths can change. Node software tells you what is running. Cluster-version evidence tells you which compatibility boundary has been crossed. Application acceptance tells you whether that state serves the shop.