Files you can
name and fetch.

A key looks like a path

In an Amazon S3 general-purpose bucket, a key identifies each object. The key photos/mug-v1.jpg looks like a file path. The slash is part of its name: a prefix lets us list names beginning with photos/, but it does not create a directory that behaves like a filesystem directory. S3's key model explains that distinction.

The service knows the object's bytes and metadata. It doesn't know that the mug is blue or that its price should appear beside it. We keep those searchable properties in the product database, which can answer a question such as “show blue products under 25 credits.” Fetching each photograph to answer that question would be an expensive way to rediscover information we already have.

To change the photo, one useful approach is to upload photos/mug-v2.jpg and then change the product record to name it. Someone still looking at the old page can finish fetching v1. New pages use v2. Eventually we can remove the unused photo, provided we know no reader still needs it. The naming convention is ours; the object store does not perform that coordination.

A whole object can be a useful boundary

Consider an ordinary object PUT: the application supplies a replacement object's contents, rather than changing a database field inside the stored bytes. That suits photographs and completed exports. It is awkward for a counter that changes every time someone visits the page. A service may offer more specialised operations, so check its API before treating “object storage” as a promise that every product behaves identically.

A large upload need not travel as one uninterrupted request. S3's multipart upload lets a client send parts separately and retry a failed part. Completing the upload assembles the object. Until completion, those uploaded parts are not the completed object a normal GET would return. This is a transfer mechanism, not a way to turn each uploaded part into a database row. See the multipart upload lifecycle.

For the same S3 key, a concurrent reader gets the old object or the replacement, not an intermediate mixture of their bytes. S3 also documents strong consistency for object reads following successful writes. Those guarantees concern the service itself; a browser or CDN may still serve a cached response. More importantly, a guarantee about one key does not make several keys change together. S3's consistency model explicitly separates these cases.

The latest files can still disagree

Each night, our shop exports prices and stock counts into separate files. Yesterday's export says the mug costs 20 credits and there are four in stock. Today's says 24 credits and eight in stock. A consumer expects the two files to describe the same export.

Suppose we replace prices.json first. Before we replace stock.json, a reader can receive the new price and old stock. Nothing stale was returned by the store: those really are the latest values of the two keys. We just made the intermediate state visible.

Instead, we can give each export its own names. Upload both B/prices.json and B/stock.json while leaving generation A untouched. Once both uploads succeed, replace one small object, current.json, that names the pair. That object is the manifest: a list telling readers which files form this export.

A reader fetches the manifest once and uses the names in that copy for the whole read. Before publication, it gets A; after publication, it gets B. Because we retain those generation files unchanged, a reader that started with A can finish after B is published. This is an application protocol built from single-object operations. It depends on readers following it and on us keeping the old files long enough.

The unfinished B files may exist in the bucket before publication. Readers must follow the manifest, rather than listing everything in the bucket and guessing what belongs together. A failed writer can leave unused files behind, so the system also needs a way to distinguish abandoned work from files a reader might still be using.

Who gets to publish next?

Now two export jobs overlap. Each begins with manifest A. One publishes B, then the other publishes C based on the same old view. If the second job must incorporate the first job's work, an unconditional replacement is not sufficient.

S3 supports a conditional write using If-Match: the replacement succeeds only if the current object's ETag still matches the one the writer read. A job that loses that check must re-read and decide whether to rebuild or retry its work. Treat the ETag as the value required by this API, not as a universal content hash. The conditional-write documentation covers the check and failure cases.

That check can protect our manifest's publication point. It still doesn't provide foreign keys, joins, or a transaction over arbitrary object updates. A table format or database built on object storage adds metadata and commit rules to supply its own higher-level behaviour. “Uses object storage” tells us where the bytes live, not all the guarantees of the system above it.

Keep the interface in mind

Block storage presents addressable blocks to a machine; a filesystem builds files and directories over storage; object storage exposes named objects through an API. These are interfaces with different responsibilities, not three grades of durability. A database can keep its files on a block device, or be designed around objects, while still exposing SQL to its users. This comparison of the three interfaces is useful background.

For our photos and exports, named objects are a good fit. The application can tolerate separate requests, it mostly writes complete assets, and it has clear names for fetching them. We still need to account for request charges, transfer, retention, and access patterns. A million tiny fetches may be a poor arrangement even when the total number of stored bytes is modest.

For checkout, the price, inventory, and order rules still belong in a system that can enforce the needed transaction. For shared file access, an application that expects filesystem operations needs those semantics too; putting a filesystem-looking interface over objects does not remove the differences.

Finally, keeping an object successfully does not prevent an authorised application from deleting it. S3 versioning can retain earlier versions and use delete markers, but storage and cleanup policies still matter. Versioning is one recovery tool; it is not the same as an independently protected backup. S3's versioning guide explains what a versioned delete preserves.

Try extending the export to include a third file of discounts. The publication boundary can still be one manifest, provided it names all three completed, immutable files and readers keep using that one manifest. The hard part has moved from replacing bytes to deciding when a coherent set is ready and when its old versions can safely disappear.

Return to the database field guide →