Failure, asynchrony and races in sequence diagrams

The deep dive, Sequence diagrams, draws Mina’s repair booking as it goes when everything works. This article draws what happens when it doesn’t: a reply that never comes, a retry that sends a second email, a worker that crashes between two arrows, a message that arrives late or twice. A happy-path diagram leaves all of these out, and a reader who builds from it assumes they can’t happen.

The arrows are the ones the deep dive taught, and so is its rule about order: a diagram promises only the order down each lifeline and from each sending to its receiving. Each section takes one way an interaction goes wrong, draws it, and says what the drawing has to show. Which of these diagrams a feature needs comes down to one test: draw a failure when it changes what someone sees or what the data ends up holding. The failure walk at the end finds them.

The examples continue the repair service: Mina’s booking R17, the notification job J9, the notification worker and the email provider. The scheduling events, the booking saga and repair R42 are invented for this article, as are all names, ids and timings.

Lost messages and timeouts

A lost message is one that is sent but never received. UML draws it as an arrow that ends in a small filled circle instead of at a lifeline.

Take the worker’s POST /emails. The request can be lost on the way out, or the provider’s reply can be lost on the way back. Or the provider can be slow, and its reply arrives after the worker has stopped listening. To the worker, all three look the same.

All the worker has is a timeout, a limit on how long it waits before giving up. Here the limit is 10 seconds. This animation keeps only the worker and the provider, and adds Mina’s inbox, which the worker can’t see. Try all three cases and watch the worker’s bar and status in each.

The worker’s bar and status are the same every time. It sends, waits 10 seconds and gives up. Mina’s inbox is not the same. When the request is lost, she gets nothing. When the reply is lost, or the provider is slow, she gets the email.

A timeout records only that 10 seconds passed. It doesn’t say whether the email was sent. Draw it as a note on the caller’s lifeline, such as “Timeout after 10 s”. Don’t draw a reply arrow labelled “failed”: nobody sent one.

An error reply is different. If the provider answers 503 Service Unavailable, that is a reply, so draw it as one. Only silence is drawn as a timeout.

A 5xx still doesn’t prove that nothing was done, because a server can fail after it has acted. The clearest case is a 502 or 504 from a gateway in front of the provider: the gateway gave up, and the provider behind it may have finished the work. Check the provider’s documentation before relying on a 5xx.

Mermaid has no arrow that ends in a circle, so its cross arrow, -x, stands in for a lost message by convention. Write the limit on the drawing even when the code relies on a library default, which can be minutes long.

Nested timeouts

The deep dive’s booking held waits inside waits: the browser waits for the API, and the API waits for the database, as its activation bars showed. Each wait can have its own limit, and the limits have to fit together. If the outer caller gives up before the inner work finishes, the work carries on after nobody is listening.

Figure 2 is that booking with a 3-second database write and one change: the browser gives up after 2 seconds, as the note at the top of its lifeline says. Watch the browser’s bar and the clock. Then set the database to 500 ms.

sequenceDiagram
    participant UI as Browser
    participant API as Repair API
    participant DB as Repair DB
    Note over UI: Gives up after 2 s
    UI->>API: POST /repairs
    API->>DB: INSERT repair request
    Note over UI: Timeout after 2 s. Shows an error.
    DB-->>API: COMMIT ok, at 3 s
    API-->>UI: 201 Created: R17
    Note over UI,API: Nobody is listening. R17 is stored.

The browser shows Mina an error at 2 seconds. The write commits at 3 seconds, and the API sends 201 Created for R17, which no browser is waiting to receive. The booking exists, and Mina was told it failed. She tries again, and now there are two bookings.

With a 500 ms write the limit is still there, but it never fires.

Each caller’s limit has to be longer than the longest its callee can take, counting the callee’s own waits. The browser waits 2 seconds, so the API should give up on the database sooner, early enough to answer while the browser is still listening. Set the database back to 3 s, and set API gives up after to 1.5 s.

The API stops waiting at 1.5 seconds and answers 503 Service Unavailable, which reaches the browser inside its 2-second limit. The browser gets an answer instead of silence, but the uncertainty remains. Nothing tells the database, so the write still commits at 3 seconds, and Mina is again told the booking failed when it didn’t. A limit set in the database itself, such as PostgreSQL’s statement_timeout, would roll the write back instead, but the reply saying so can be lost too.

What makes her second attempt safe is a key on the booking request, like the one on the email in the next section. The API stores the key with the booking, in the same transaction, so a retry with the same key finds R17 instead of making a second booking.

Retries and idempotency keys

A retry sends the same request again, unchanged, after a failure or a timeout. For the worker it is the obvious next step. Whether it is safe depends on what happened the first time.

Start with the reply lost, and count the emails in Mina’s inbox. Then switch the first attempt to Request is lost.

sequenceDiagram
    participant Worker as Notification worker
    participant Provider as Email provider
    Worker->>Provider: POST /emails, Idempotency-Key: repair-email-J9
    Provider--xWorker: 202 Accepted
    Note over Worker: Timeout after 10 s. Wait 2 s.
    Worker->>Provider: POST /emails, Idempotency-Key: repair-email-J9
    Note over Provider: Key seen before: send nothing, return the saved reply
    Provider-->>Worker: 202 Accepted

When the request was lost, the retry is harmless and Mina gets one email. When the reply was lost, the provider had already sent the email, so the retry sends a second copy. Every retry therefore risks a duplicate, unless the provider can recognise a repeat.

An operation is idempotent if doing it twice has the same effect as doing it once. Sending an email isn’t. An idempotency key lets the provider make it idempotent. The worker attaches a key that names the operation, here repair-email-J9, and sends the same key on every attempt. The provider remembers keys it has seen, and a repeat gets the saved reply without a second email.

Derive the key from the job, not from the attempt. A new random key on each retry tells the provider that each attempt is a new email.

Now set Idempotency key to Sent and play both cases again. Mina gets one email in each. The key protects only while the provider remembers it: Resend, an email API, keeps keys for 24 hours, so a retry 25 hours later sends a second email. Retries should also wait longer after each failure, with a random amount added so that workers don’t all retry at once, as Amazon’s guidance on timeouts, retries and backoff with jitter explains.

A drawing doesn’t have to show every attempt. Draw them out when what differs between attempts is the point, as Figure 3 does with the key sent: the second attempt carries the same key as the first. When the retry is policy, write a loop labelled “up to 3 attempts”, or a note such as “Retried up to 3 times with backoff”, and keep the picture for what you want discussed.

Retries also stack. An HTTP client that retries three times inside a worker that retries three times, inside a queue that redelivers three times, can make 27 attempts at the provider. Draw a layer’s retries when they multiply like this, and leave them out when they don’t.

Failing part-way

A crash between two arrows leaves whatever the first arrow did and none of what the second would have done. Every such gap is a space between two rows, so the diagram is where to look for them.

Figure 4 is the deep dive’s outbox with its failures switched on. Set Provider down, then Worker crashes.

With the provider down, the browser’s part of the diagram doesn’t change at all. The worker records that J9 should be tried again in 30 seconds, and the job waits in the database.

With the worker crashing, the provider sends the email and the worker dies before it marks J9 sent. Five minutes later another worker claims J9, and the provider sends a second email. The five minutes are the claim’s lease. A worker claims a job with a locking read, so only one worker gets it. That lock ends when the claim commits, long before the email goes out, so the claim also marks the job as taken for five minutes, and a dead worker’s job comes free again. The lease has to outlast the worker’s longest attempt, or a second worker takes a job the first is still sending. The new worker can’t tell that the email already went. The idempotency key closes this gap too: the second worker sends repair-email-J9 again, and the provider recognises it.

The outbox exists to close a gap of the same kind. If the API stores the booking and then publishes J9 to a queue as a separate step, a crash between the two leaves a booking with no email. Publishing first can send an email for a booking that rolled back. Drawn as separate arrows, the gap is a row you can point at:

sequenceDiagram
    participant API as Repair API
    participant DB as Repair DB
    participant Queue as Email queue
    API->>DB: INSERT booking R17
    DB-->>API: COMMIT ok
    Note over API: A crash here: booking, no email
    API->>Queue: Publish J9
    Queue-->>API: Stored

When the steps belong to different services, no transaction spans them, and a failure part-way has to be undone with new messages. A design built this way is usually called a saga. Figure 5 books an appointment in three services, and Billing declines the deposit at the end. Play it, then set Deposit to Accepted to compare the run with nothing to undo.

Draw the undo as what it is: new requests with their own replies, sent after the failure. Nothing was rolled back. Between the hold and its release, A5’s time was taken and anyone asking for it was refused. In the third case inventory’s reply to the release is lost, so the release is sent again and has to be safe to repeat. A step that can’t be undone, such as an email sent, goes after everything that could still fail.

The figure is orchestrated, the fork shape from the deep dive’s Fork and stair: one booking service sends every command. A choreographed saga is the stair, with each service reacting to the last one’s event, and the whole booking exists only as that chain of arrows. How to design either belongs to messaging, a field this site hasn’t covered yet.

Brokers and acknowledgements

Put a queue in place of the polling worker. The API publishes J9 to an email queue, and a worker later receives it. Whether the queue gets a lifeline is the framing choice in the deep dive’s Choosing what to draw. Here it gets one, because what it does after a crash is the question. Draw the publish as a call with a reply only when the API waits for the broker to confirm it has stored the message, since not every client waits.

A worker that takes J9 does three things: receives it, does the work, and tells the queue it is finished. Draw them as three messages. The queue keeps J9 until it hears the third, and offers it again if the acknowledgement doesn’t come in time. Play the first case, then the other two.

With the acknowledgement after the work, a crash between the email and the acknowledgement leaves J9 unfinished, and a second delivery sends a second email. It is Figure 4’s crashed worker with a different holder, and the idempotency key is the remedy again.

With the acknowledgement before the work, the crash loses J9. Nobody gets a duplicate, and nobody gets the email either. The usual choice is to acknowledge after the work and make the work safe to repeat. Drawing the acknowledgement as its own arrow makes that choice visible. A diagram that leaves it out implies the message was handled the moment it arrived.

What the broker promises can’t be read from the arrows, so write it in a note with the product named: how long it waits for the acknowledgement, and whether a message can arrive twice with nothing gone wrong, as Amazon SQS standard queues say it can. A diagram that ends at a dead-letter queue claims that someone reads it, so name who.

Fan-out to several consumers

When an appointment is scheduled, the scheduling service publishes one event. The technician app needs it to plan a route, and billing needs it to raise the deposit invoice. Start the figure, then switch to Billing is down.

The arrows to the subscribers leave the topic, not the scheduling service, because the publisher doesn’t know who listens. Keep the list of subscribers in a note, where it can change. The deliveries have no order between them, which the par frame says, so Billing answers first is an equally valid history.

Failure is separate for each subscriber. With billing down, the technician app acts on A5 before billing knows about it. Whether billing’s copy waits for it depends on how its subscription is set up, so the figure assumes a durable one and says so in a note. If billing must have the deposit before the appointment is confirmed, this is not a fan-out, and the flow needs a reply from billing.

Two workers reading one queue look the same on the page, two arrows leaving a broker lifeline, but each message goes to only one of them. A note has to carry that difference.

Work that finishes later

A result that takes longer than a request can come back in three ways. The API answers 202 Accepted and the caller polls a status resource, as in the deep dive’s polling loop. The other system calls back later. Or a stream of updates arrives over one open connection, drawn as a loop of asynchronous sends. In the drawing, the difference is who sends the last arrow.

A callback is a new message

A webhook is a request to us made by another system, later, because of something we asked for earlier. The email provider accepts J9 now and reports a bounce hours afterwards. Drawing the bounce as a reply to J9’s request is the most common error in diagrams of these flows. The bounce is a fresh request from the provider to a different participant, the Repair API’s bounce endpoint, with its own reply. A dotted arrow from the provider back to the worker, labelled “bounced”, would say that the worker was still waiting hours later. It had long since finished.

Draw the callback as a solid call to the receiving participant, followed by our reply. That reply, 200 OK, means “received”, not “handled”. The only thing that connects the bounce to J9 is an identifier, a correlation id, and three arrows carry it. The provider returns its id, m-481, in the 202. The worker stores it against J9. The bounce carries it back, and the receiver looks J9 up by it. Postmark’s bounce webhook works this way, with the original email’s MessageID in its payload.

The callback that arrives first

The worker’s save and the provider’s callback are on different lifelines, so nothing orders them. In the usual case the save wins by hours. Occasionally the email bounces within seconds, and the callback reaches the API before the worker has written m-481 down. Play all three.

In the second case the API looks for a job with m-481, finds none and answers 200, as it would for any callback it had handled. The provider has been told the bounce was received and won’t send it again. J9 never learns it bounced.

The third case stores the bounce as it arrived, answers 200 and matches it later, once the worker has saved the id. Providers that retry can also deliver the same callback twice, so a note on the receiver, such as “skip bounce m-481 if already recorded”, belongs in the drawing.

Long waits and missing callbacks

No bounce arrived, and that proves less than it seems. A provider that reports only failures says nothing when the email is delivered, and nothing when its own report is lost. J9’s honest status is “sent, delivery unconfirmed”, which a note can say. If the flow needs a positive result, such as a payment confirmed, the missing callback needs a plan: a timer that fires after a day and asks the provider what happened. An absent arrow is the hardest thing in a diagram to notice, so write the plan down.

A wait that lasts hours or days needs three things on the page. The first is the gap, in a note across every lifeline, such as “24 hours later”. The second is what holds the flow while it waits, such as a row “A5 held until Friday 10:00”, because the process may restart many times. The third is what wakes it, usually a scheduler with a lifeline of its own, as the deep dive’s Flows with no caller describes. Split such diagrams at the waits. When you find yourself drawing a fourth variant of the same wait, the question has become which events each condition allows. States and lifecycles draws that in one diagram, and Waiting, reminders and cancellation draws who owns the wait.

Races and overtaking

The deep dive’s order rule says two events on different lifelines have no order unless a chain of messages connects them. With asynchronous messages that limit becomes the point of the diagram.

Mina moves A5 to 14 October, then cancels it. Scheduling publishes Moved (version 2) and then Cancelled (version 3). What billing receives is drawn on the billing lifeline. Start with the default, then set billing to check the version.

Billing that applies what arrives ends up with an invoice for a cancelled appointment. The cancellation came first, the move arrived last, and billing treated the move as the latest news. The drawing shows this because the two messages leave the topic in the opposite order to the one they entered. Put a version or sequence number in the labels, as here, or the reader can’t see the overtaking.

What order a design can rely on depends on the broker and how it is used, so write it in a note. Kafka keeps the order of events with the same key, so “ordered per appointment id” can be a claim about a design with a product behind it, never a property of the notation. Even then a redelivered message arrives after later ones, so a consumer that matters should defend itself. It can keep the highest version it has seen, as in the figure, or ignore the event’s content and fetch the current state from its source.

The same open order produces the stale read. A page that lists repairs from a table updated by a consumer can read before the update, because the read and the update sit on different lifelines. Say which remedy the design relies on, such as reading from the record the write went to, or waiting until the consumer reports that version. Otherwise the reader will assume the page always sees the new row.

It also lets two requests race for one record. Draw the bad interleaving as a diagram of its own, with a lifeline for each instance, so the reader can see it. Facebook’s memcache paper describes a real one, the stale set, where a slow reader writes an old value into the cache after a newer write has invalidated it. The practice at the end works a race through.

The failure walk

Every failure in this article was found the same way: take one arrow and ask what happens if it goes wrong. Walk each arrow of a happy-path diagram with these questions, and draw a diagram for each answer that changes what someone sees or what the data holds.

• Lost. The request never arrives, or the reply never comes back. Who is waiting, for how long, and what does each side know afterwards? Silence is a timeout note on the caller, never a reply labelled “failed”.
• Late. The reply arrives after the caller gave up. Is each caller’s limit longer than what its callee can take?
• Twice. A retry or a redelivery repeats it. What makes the repeat safe, and how many layers retry?
• Out of order. Which events are drawn in an order that nothing guarantees? Does a par section need another section’s reply, or a consumer need events in the order they were sent?
• Receiver down. Who holds the message until it returns? For every asynchronous message, who notices if it fails for good?
• Crash between two arrows. Where does it leave the data?

Try three of them on the outbox diagram. Who notices if the email fails? The worker, which records that J9 should be tried again in 30 seconds. The browser never learns of it. Where does a crash between two arrows leave the data? Between the provider’s 202 and “Mark J9 sent”, the email is out and J9 is unmarked, so the claim expires and a second email goes unless the key stops it. Which events are drawn in an order nothing guarantees? The browser’s 201 and the worker’s claim.

A question the drawing can’t answer has found a gap in the design or in the picture. Both are worth finding before the flow is built.

Drawing what happened

To reconstruct an incident, start from evidence rather than the design. Logs and distributed traces record which calls were actually made and how long each took. A trace’s waterfall view is a timeline of one request, not a sequence diagram, but it tells you which messages to draw and gives the real time the diagram leaves out.

Order the events by what caused what, not by their timestamps. Clocks on different machines disagree, so a log line stamped earlier on one host can describe an event that happened after one on another. A request comes before the handling it caused, and one process’s log is in order from top to bottom. That is the order rule applied to evidence. A trace gives those links directly, because each span records the span that caused it.

Draw what happened, then compare it with the design diagram to see where the two disagree. A tool that generates a sequence diagram from a trace draws every call, so edit the result down to the messages that answer your question, as you would any other diagram.

Practise with a cancellation race

R42 is Approved. At almost the same moment, Mina cancels it in her browser and a technician starts the repair in an app. Two instances of the Repair API handle the two requests, and both use Repair DB. Draw one sequence diagram of this. Decide which participants need lifelines, what each person is told, and what the database does when both requests want to change the same row. Write down what Mina and the technician each see before you read on.

sequenceDiagram
    participant Mina as Mina’s browser
    participant Tech as Technician app
    participant A as API instance A
    participant B as API instance B
    participant DB as Repair DB
    par cancel
        Mina->>A: POST /repairs/R42/cancel
        A->>DB: UPDATE status = Cancelled WHERE status = Approved
        DB-->>A: 1 row changed
        A-->>Mina: 200 Cancelled
    and start
        Tech->>B: POST /repairs/R42/start
        B->>DB: UPDATE status = In repair WHERE status = Approved
        DB-->>B: 0 rows changed
        B->>DB: SELECT status
        DB-->>B: Cancelled
        B-->>Tech: 409 Already cancelled
    end
    Note over DB: Check and change are one statement. The other order is equally valid.

Sources and further reading

• Marc Brooker, Amazon Builders’ Library, Timeouts, retries and backoff with jitter: choosing limits, why retries multiply and how to space them out.
• Resend, Idempotency keys: one provider’s keys and how long it keeps them.
• Chris Richardson, Pattern: Saga, and the Azure Architecture Center’s Compensating Transaction pattern: orchestration and choreography, compensation that can fail, and points of no return.
• Amazon SQS, Standard queues: at-least-once delivery and best-effort ordering, as an example of what to look for in any broker’s documentation.
• Apache Kafka, Introduction: keys, partitions and the order in which a consumer reads a partition.
• Postmark, Bounce webhook: a callback that carries the original message id.
• Rajesh Nishtala and others, Scaling Memcache at Facebook (NSDI 2013), section 3.2.1: the stale set and the leases that prevent it.
• On this site: Sequence diagrams, the deep dive, for the notation, the order rule and the outbox this article builds on.
• On this site: States and lifecycles and Processes and swimlanes, for the views to turn to when a sequence stops being the right drawing, and Handling conflicts and uncertain outcomes, for retries on the database side.