event-driven architecture
why async exists, what it buys, and the guarantees you have to build yourself.
why async exists, what it buys, and the guarantees you have to build yourself.
components that communicate by producing and consuming events, instead of calling each other directly. the producer does not know who is listening. that is the whole idea, and every benefit and every cost follows from it.
the pattern itself is simple. what is worth understanding is which problems push you toward it, and which guarantees you stop getting for free.
in a synchronous system, A calls B and waits. simple, debuggable, and tightly coupled in three ways: A cannot proceed until B responds, so B's latency is A's latency and B's outage is A's outage. A must know B's address and contract. and adding a third consumer means changing A.
that third one is where it usually breaks down. an order is placed, and now billing, inventory, notifications, analytics, and fraud detection all care. calling five services in sequence means A's latency is the sum of five, A fails if any of them fail, and the sixth consumer means editing A again.
with events, A appends "order placed" and moves on. the consumers subscribe.
temporal decoupling. A returns immediately. consumers process at their own pace, and a consumer being down means a growing backlog rather than a failed order.
additive extension. a new consumer subscribes. the producer is not modified and does not know.
a history you can replay. if events are retained, you have an ordered record of what happened. useful for debugging, auditing, and rebuilding derived state from scratch, which is a genuinely powerful recovery tool.
the guarantee you gave up is the one synchronous calls provided implicitly: you no longer know whether the work happened.
A emitted the event and returned success to the user. billing may have failed. inventory may have processed it twice. notifications may have sent the wrong thing. A has already told the user it worked.
recovering from that is not a detail. it is most of the engineering:
idempotency, because delivery is at-least-once. exactly-once delivery across a network is not achievable in general; what systems offer is at-least-once plus deduplication. kafka's exactly-once semantics are real but narrow. they cover consume-transform-produce within kafka, not the side effects your consumer performs against a payment API. once a consumer touches anything outside the log, idempotency keys are on you.
eventual consistency as a user-visible property. the order exists and the invoice does not, for some window. someone has to decide what the UI shows during it.
compensating actions. there is no rollback. undoing a partially completed workflow means explicitly modelling the reversal of each step.
failure has to go somewhere. a consumer that cannot process an event will retry forever and block the partition behind it unless there is a dead letter queue, and a DLQ nobody monitors is a silent data loss channel that looks like success.
debugging is a different activity. a synchronous failure is a stack trace. an async failure is correlating a causation ID across services and reconstructing a timeline from logs that do not share a clock. distributed tracing is not optional here. without it, "the invoice never appeared" is close to uninvestigable.
event schemas are API contracts, and worse ones. once consumers depend on a field you cannot change it, and unlike an HTTP API you often cannot enumerate your consumers or force them to upgrade. adding fields is safe. removing them breaks consumers. changing the meaning of a field breaks them silently, which is the one that reaches production. this needs a schema registry and compatibility rules from the start, not after the first incident.
ordering is per-partition, not global. two events for the same entity can be processed out of order unless they share a partition key. get this wrong and "cancelled" is applied before "created". the fix, key by entity ID, is easy; noticing you needed it is the hard part.
durability is a setting, and its default is not what you want. kafka's acks controls when a produce is acknowledged:
acks=0 # fire and forget. the broker may never have received it.
acks=1 # the leader has it. a leader failover here loses the message.
acks=all # every in-sync replica has it.acks=all only means what you want alongside min.insync.replicas. with min.insync.replicas=1, "all in-sync replicas" can be a single replica, and you have acks=1 with extra steps. the replication design docs cover why the usual production pairing is acks=all with min.insync.replicas=2 on a replication factor of 3. that tolerates one broker loss without either losing writes or blocking them.
events are a good fit when one thing happening should trigger several independent reactions, when the producer genuinely should not wait for the consumers, when consumers must be able to be down without the producer failing, or when you need an auditable history of state changes.
they fit badly in the opposite cases, and the mismatch is usually one of these three. a user clicking "buy" needs to know whether it worked, and "your event was accepted" is not an answer. a single consumer that always processes exactly one producer's events does not need a broker between them. that is an HTTP call with more infrastructure. and if the result must be consistent across services before the caller returns, eventual consistency is not a tradeoff you can make.
the log, kafka, kinesis, pulsar, or a database-backed queue, is infrastructure with its own operational surface, and its configuration decisions are long-lived.
retention determines whether replay is possible at all; retention.ms set to seven days means the audit trail you were relying on is a seven-day window. partitioning determines both parallelism and ordering, and repartitioning a live topic changes which key lands where, which breaks ordering guarantees during the migration. consumer group semantics determine what happens when a consumer is slow, and lag that grows faster than it drains is a queue that never recovers.
none of these are configuration details you tune later. they are architectural commitments that are hard to change once producers and consumers depend on the current behavior, which is the real cost of choosing event-driven, and the one least likely to appear in the design doc.