Context, problem, and forces

Our cloud-native, reactive systems are composed of bounded isolated components, which provide proper bulkheads to make the components responsive, resilient, and elastic. The isolation is achieved via asynchronous, message-driven, inter-component communication. Components communicate by publishing events to an event stream as their state changes. We have chosen to leverage value-added cloud services to implement our event streaming and our databases. This empowers self-sufficient, full-stack teams to focus their efforts on the requirements of their components and delegate the complexity of operating these services to the cloud provider. Modern database and messaging technology has abandoned two-phase commit (2PC) distributed transactions in favor of eventual consistency. 2PC does not scale globally without unrealistic infrastructure costs, whereas eventual consistency scales horizontally to support systems of all scales. Event streaming is the mechanism for implementing eventual consistency.

In an eventually consistent system, we need an approach to atomically publish events to the stream as the state in the database changes, without the aid of 2PC. We also need a historical record of all the events in the system for auditing, compensation, replay, and resubmission. Furthermore, in high volume, highly contentious systems, we need an approach for atomically persisting state changes.

The state or data in a system for all practical purposes originates with user interactions. At the beginning of any value stream, data must be entered into the system. Traditionally, this CRUD interaction only stores the current state of the data and thus the history or audit trail of the data is lost. Traditional event sourcing solutions persist the data as a series of data change events. This requires the system to recalculate the current state before presenting the data to the user, which is inefficient. Alternatively, the system can use the CQRS pattern to persist the current state in a materialized view as well, which can overly complicate the implementation of usually straightforward CRUD implementations. We will instead use CQRS at the opposite end of the value stream, as discussed in Chapter 4, Boundary Patterns. Furthermore, the traditional approach for turning these event records into events on a stream still requires 2PC to flag event records as published.

Downstream components in a chain of interactions need to capture events, perform some logic on the events, and store the results. For high volume processing, we want to avoid the contention of updates, which causes errors and increases retries and thus impedes throughput. We also need to recognize that event streaming only guarantees at-least-once delivery and that the events are delivered in the order in which they were received. We will see in the Stream Circuit Breaker pattern that there are various reasons that events will be delivered more than once and that the order received is not always the order that is important for the business logic. The ACID 2.0 transaction model (Associative, Commutative, Idempotent, and Distributed) recognizes these realities and can be combined with event sourcing in downstream components to implement logic that is tolerant of these conditions.