Time, snapshots and distributed reads

One snapshot across shards

Start with a report that checks whether order 103 exists, then reads P7’s stock. If the purchase commits between those checks, A can answer “no order” and P “stock 0”. Each answer was current when made. Taken together, they miss both the state before the purchase and the state after it.

Give the purchase a shared commit timestamp, 40. These numbers identify positions in the example’s history; they are not seconds or order dates. A report that reads A at 35 and P at 45 straddles the purchase. Choosing 35 for both owners, or 45 for both, instead selects one side of its commit boundary.

At 35, neither part appears. At 45, both do. Reading at 45 does not require the two machines to respond simultaneously. A can answer first and P later, provided both select versions for 45 and can prove those versions form the required history.

In this timestamped model, the read takes the latest committed version of each key whose timestamp is at or below its chosen time. A transaction’s related writes share the commit boundary, so a snapshot cannot include its stock decrement while excluding its order. A missing row is an answer too: A must know that order 103 was absent at 35, rather than simply not having received it yet.

The timestamp is part of a transaction protocol. Independently stamping rows with local clocks does not create atomic visibility. Transactions across partitions follows the joint commit decision and its recovery. Here we assume that decision and ask when a reader can safely observe it.

Timestamps and logical clocks

A’s local log entry 8 and P’s local log entry 8 need not describe the same point in the purchase. The logs contain different work. Their positions order each shard’s own history; comparing the numbers across shards supplies no common snapshot.

A shared timestamp system needs an ordering rule. A logical clock is a counter advanced at local events and carried in messages. Before sending, a node increments its counter. On receiving a message, it sets the counter above both its own value and the received value. Suppose A starts at 7 and P at 3.

A sends 8. P receives it and advances to 9, then to 10 at its next local event. The message and everything following it at P now have timestamps greater than the send. This preserves causal order: when an event can influence another through local order and messages, its timestamp is smaller.

The reverse inference does not hold. An unrelated node can reach 20 without hearing from A or P. Its larger number proves no causal connection, and a counter reaching 45 says nothing about whether the data needed for snapshot 45 has arrived.

A hybrid logical clock, or HLC, combines a physical time component with a logical counter. Compare the pair by physical component first, then logical component. Suppose P starts at (98, 0). Its physical clock still reads 98 when it receives (100, 0). So it moves to (100, 1), beyond the message. When its physical clock later reaches 103, it can move to (103, 0).

The physical component keeps the timestamp related to clock time while the logical component distinguishes events without making time run backwards. It still needs the database’s version, concurrency and commit rules. HLC is useful ordering machinery; it is not by itself a proof of serializable execution, a real-time freshness promise or a ready replica.

Clock uncertainty and real-time order

A consistent view can be older than the caller expects. Suppose a write finishes on one node before a read starts on another, but the writer’s clock is ahead. The actual event order and the local clock readings can disagree.

The write completes at actual time 1,000 ms, with the writer’s clock at 1,008. Two milliseconds later, the reader’s clock says 998. If those local readings alone chose the versions, a read at 998 could omit the completed write. The resulting past view might be internally consistent while failing a promise to include transactions completed before the read began.

Strict serializability adds this real-time order to serializable transactions: when one finishes before another starts, their serial order must agree. A deliberately historical read instead asks for an earlier point. It can be entirely correct to omit the recent write under that contract. Choosing a consistent past and promising a current observation are different requests.

Spanner’s published design uses TrueTime, which reports an interval containing the current time. Its commit protocol chooses a timestamp and waits until even the earliest possible current time has passed it before reporting success. A later operation can then be ordered after the acknowledged timestamp. This relies on the bounds being valid, alongside the transaction and replication protocol; the interval alone does not commit anything.

An HLC system can handle uncertainty at reads as well. CockroachDB detects values above a read’s timestamp that could have been written earlier because of clock skew. It may move the transaction forward, validate earlier reads, or retry when validation fails. A larger timestamp cannot simply be dismissed as a future write. The exact isolation and retry behaviour belongs to CockroachDB isolation and follower reads.

Neither approach makes clock assumptions free. Wider uncertainty can add waiting or retries; violating an assumed bound can invalidate guarantees that depend on it. The useful question is what the database promises for this operation, and how it establishes that promise.

When a replica can answer

Return to snapshot 45. P’s follower has a new clock reading and an older copy of the data. It cannot infer that the absence of a write means no write belongs before 45. A delayed message or an unresolved transaction may still change the answer.

To serve the chosen time, the target needs a complete relevant history through it and the state needed to read that history. One way to establish completeness is a closed timestamp: an authority’s promise that no new write will be introduced at or below a bound. The promise must be tied to the history that the follower has to apply.

Suppose P’s follower is asked to read at 45, but has applied its log only through position n6. The authority’s closing promise for 45 requires history through n8. The timestamp and the log position name different things: the promise identifies the read boundary, while n8 identifies the commands needed to serve it.

Receiving that promise does not supply the missing state. Even after the entries through n8 are durable, application may remain at n6: the follower could recover them after a crash but cannot yet answer this read. It qualifies only once it has both the closing promise and the required applied state. A must meet the same requirement for its own portion; P qualifying does not qualify A.

This model uses a complete frontier that also accounts for relevant unresolved outcomes. A provisional write at 38 cannot be guessed absent just because its commit decision is missing. A system may wait for that decision, resolve it through transaction records, or move the write beyond the read boundary under its protocol. A raw closure notification need not mean every intent has already been cleaned up.

CockroachDB’s closed timestamp is associated with a required position in the replicated log. A follower can use the promise after applying the required commands. Spanner’s published safe-time design similarly accounts for replication progress and prepared transactions. These are concrete ways of proving completeness, rather than properties obtained from reading a clock.

A historical read can therefore wait or fail on a reachable machine. Its required state or outcome may be unavailable. Choose a bounded wait, a route to another qualified replica, or an explicit unavailable result. Returning whatever state happens to be local would change the read’s promise. Replicas, lag and read routing follows the routing and session requirements for one copied history.

Choosing a historical snapshot

An older snapshot can be easier to serve because more replicas have caught up to it. It also requires older versions. A replica’s serviceable history therefore has two ends: the oldest history still available for the read, and the newest history it can prove safe and apply.

Suppose A can serve times 20 through 60 and P 30 through 50. A common snapshot must lie in their intersection, 30 through 50. Each target’s retained history and safe applied frontier constrain the same chosen time.

Both 35 and 45 fit the original overlap, though they give different purchase states. Time 55 asks P to go beyond its proof. Waiting or rerouting might make that time available. Time 25 asks P for history it may already have collected. Applying newer log entries cannot bring those collected versions back. Another retained copy or recovery source would be a separate operation, rather than a replay wait.

When A retains only from 52, the two targets have no common time: A starts after P ends. Choosing A’s oldest time and P’s newest time separately would hide that failure by mixing snapshots. The correct result is to find other qualified targets, wait where waiting can help, or report that this request cannot be met.

An exact historical read fixes one timestamp for the whole observation. A bounded-staleness read permits the system to choose within a freshness limit. Suppose the task allows only timestamps at or after 48. The original overlap narrows to 48–50. Requiring at least 55 leaves no eligible point on these targets.

Those last numbers are positions in this model, not a seconds-based lag limit. A time-based bound such as “at most five seconds old” additionally needs a trustworthy relationship between timestamps and current time. Native APIs also differ in which queries and transaction scopes support each choice. Fix the shared observation before assuming that separate requests using the same relative age will pick the same timestamp.

A report split into several queries or pages must retain its chosen snapshot across that whole unit of work. Recomputing “a few seconds ago” at every page advances the history. A retention policy must leave the required versions available for as long as that read contract needs them; indefinite historical access is a storage commitment, not an automatic consequence of keeping versions.

A report is not a stock reservation

Snapshot 35 correctly reports one blue mug left. Purchase 103 can already have committed by the time the report arrives. Even a current stock read can be followed immediately by another buyer’s purchase. Seeing 1 establishes an observation, not an allowance to sell another mug.

The purchase must check and change stock together at its authority. The guarded decrement from Transactions and isolation expresses the condition in the write:

UPDATE products
SET stock = stock - 1
WHERE product_id = 'P7' AND stock >= 1
RETURNING stock;

If current stock is 0, this statement returns no row. If it is 1, a successful decrement returns 0. The complete purchase still needs the isolation and atomic transaction protocol covering both products, the order and its lines, with any required retry handling. A historical report cannot supply that protocol’s current authority.

The read contract names the snapshot’s scope, its permitted age and what happens when no copy qualifies. Those choices explain why a nearby follower can suit a report while the next purchase still needs coordination. For the concrete isolation modes, historical SQL and follower-read controls, continue with CockroachDB isolation and follower reads.