Designing distributed software systems: a practical introduction

A distributed application places work in separate processes that communicate over a network. That arrangement introduces failure modes in which a request is delayed, repeated or completed even though its response is lost. Timeout and retry behaviour therefore form part of the application protocol rather than remaining an implementation detail.^[1]

Microsoft describes independent scaling and deployment as microservice benefits, alongside complexity in areas such as interservice communication, testing and data consistency.^[12] Distribution can therefore address requirements such as independent scaling, fault isolation or separate release boundaries while adding remote calls, operational dependencies and cross-process data coordination. The decision to distribute a component is therefore testable against explicit requirements such as expected load, response-time objectives, recovery targets, data-loss tolerance and team ownership.

Clients pass through a load balancer to two service replicas; both use a primary database, which replicates to a read replica, while services can publish to an event queue — Application traffic and database replication are separate flows; replication runs from the primary database to the read replica.

Remote calls need bounded failure behaviour

A timeout limits how long a caller waits for a remote operation. AWS describes timeouts, retries and backoff as techniques for handling partial and transient failures, while warning that retries can increase load on a service that is already failing.^[1] Where retry is appropriate, the retry count, backoff and jitter can be selected from the operation’s latency distribution, idempotency and load limits rather than applied identically to every call.^[1]

A circuit breaker can temporarily block calls after failures reach a threshold, but Microsoft also lists situations in which retries or message-system controls are sufficient and the breaker would add complexity.^[6] A bulkhead isolates resource pools so that excessive load or failure in one area does not exhaust resources needed by unrelated work.^[7] These patterns are conditional controls whose thresholds and partitions require measurements from the system they protect.

Repeated operations and duplicate delivery

Retries and at-least-once message delivery can cause the same logical operation to arrive more than once. Microsoft recommends idempotent processing when a failed consumer may retrieve a message again, and notes that competing consumers do not guarantee creation order without additional controls.^[8] For a side-effecting API, an idempotency key can identify one logical request. Stripe documents one concrete contract that retains a result, compares parameters on reuse and prunes keys after a stated period.^[13] Other APIs need to define their own scope, lifetime and conflict behaviour.

A uniqueness constraint can enforce one stored record per idempotency key.^[10] When the protected database change and key record are part of one database transaction, PostgreSQL describes the steps as one all-or-nothing operation.^[11] A database transaction does not by itself make an external effect atomic with the database. If the workflow crosses that boundary, the design needs an explicit coordination mechanism.

Replication direction and consistency

In the corrected architecture figure, application replicas send database operations to the primary database. The database replication arrow runs from the primary database to the read replica. PostgreSQL’s standby documentation describes a primary server producing a continuous sequence of write-ahead log records and a standby consuming that sequence.^[9] An application may send eligible reads to the replica, but acceptable staleness and failover behaviour depend on the database configuration and the data’s consistency requirements.

Replication can increase read capacity or provide a standby copy, while asynchronous replication allows a period in which the replica has not yet applied a committed primary change.^[9] A catalogue description may permit that delay. A reservation, authorisation or uniqueness decision may require a stronger guarantee. The choice follows the tolerated staleness, recovery point and consequence of a stale decision.

Cross-service transactions

The saga pattern divides a transaction into local transactions. Each participant commits its local work and initiates the next step through a message or event.^[2] If a later step fails, compensating transactions may attempt to counter earlier work. Microsoft notes that compensation can itself fail and that sagas require monitoring and tracking.^[2] A saga is therefore applicable when the business process can define intermediate states and compensations; it is not a replacement for every atomic transaction.

Sequence diagram of an order saga where failed payment triggers compensation that releases an inventory reservation — A saga: less magic wand, more carefully organised bookkeeping.

The retained saga figure shows inventory reserved before payment and a release action after payment failure. Its meaning depends on the domain defining release as a valid compensation and on the workflow recording enough state to retry or investigate an unsuccessful compensation.^[2]

Database changes and events

Writing application data and publishing an event are two separate operations when they target different systems. AWS documents the inconsistency that can arise when one succeeds and the other fails.^[3] A transactional outbox records the data change and an event record in one local database transaction; a separate publisher later sends pending events.^[3]

The outbox publisher may send a message more than once, so AWS recommends that consumers track processed messages and behave idempotently.^[3] Ordering controls are required only where the domain depends on order, and the partitioning key can then represent the entity whose events must remain sequenced.

Availability and observability

A fallback is appropriate only when its result remains valid for the operation. Cached catalogue data may satisfy a defined staleness limit, while a payment or booking decision may prohibit a fallback that invents an unrecorded success. Essential and optional dependencies can therefore be classified per user journey, with each degraded behaviour tested against an explicit requirement.

Google SRE identifies latency, traffic, errors and saturation as four signals for monitoring a service.^[4] It also distinguishes black-box monitoring of externally visible behaviour from white-box telemetry used to diagnose internal causes.^[4] Metrics, structured logs and traces can be selected to answer the service-level objectives and incident questions defined for a system.

Contracts that can evolve

Consumers rely on particular parts of a provider’s interface. Consumer-driven contracts record those expectations so a provider can check changes against actual consumer use.^[5] Fowler also describes extension points and rules for handling unknown elements as mechanisms for forwards and backwards compatible schemas.^[5] Contract tests complement integration and production verification because they cover declared expectations rather than every runtime dependency.

A conditional design review

A review can begin with one important user journey and its measurable objectives. For each remote step, the review records the timeout, whether retry is safe, the maximum retry load, duplicate handling and recovery path. For each data flow, it records consistency, staleness and recovery requirements. For each asynchronous step, it records ordering scope, poison-message handling and observability. Failure tests can then check those stated conditions in a controlled environment. The timeout and retry controls can be compared with measured latency and load, while monitoring can be compared with user-visible symptoms and the four service signals.^[1]^[4]

No single pattern removes distributed uncertainty. The architecture is verifiable when late responses, duplicate work, stale replicas, failed compensation and incompatible changes have defined outcomes that can be measured against requirements.