The outbox pattern operates on a principle that acts as a temporary storage for events. The idea is not to send events directly to other systems after some operation was performed on the API, but to store them in a local temporary table, called the outbox table. Later on, those events are read from the outbox table and sent to the message broker. The key point here is that all the business logic and events are stored in the scope of a single transaction. This ensures that either all the steps inside a transaction will be executed, or none of them.
And that is the main purpose of this pattern; to ensure atomicity and consistency. When the event is sent after the service performs the operation on a database, two things can happen:
- if the database operation is successful but the event is not sent, the downstream service will not be aware of the change;
- if the database operation fails, but the event is sent, data might be corrupted.
In a typical scenario, an event flow looks like on the diagram below. After the service receives the request, some database operation is performed and the event is sent to notify other systems. After both operations succeed, the successful response will be returned.
But what happens if sending the event fails? We can rollback the operation on a database and return a failed response. But what if rollback also fails? The service will return a failed response but let’s see what happened to the data. The first operation on a database will modify the data, the event will not be delivered and the second operation on a database will not revert the changes made in the first operation. This will lead to an inconsistent state as the data will be corrupted.
Therefore those two operations should run atomically, so either all steps in the same transaction are executed or none of them is. The outbox pattern stores the events in the outbox table before sending them to the message broker. So what happens if updating the database fails or inserting the event into the outbox table fails? Since both operations are running in a single transaction, the whole operation will fail in this case. And if both succeed, the event will later be retrieved from the outbox table and sent to the message broker. If that fails, it can be retried. This example also describes the principle of a guaranteed message delivery.
From a technical point of view, the outbox pattern has a special table, called the outbox table. It serves as a temporary storage for events. Service that implements the outbox pattern uses this table along standard create, update and delete operations. When those kinds of operations are executed, the event is stored into the outbox table in the same database transaction. Events that are stored inside the outbox table are sent to the message broker, which then distributes the events to other services. For reading the events from the outbox table and sending them to the message broker, the outbox pattern uses a special component, an outbox processor. This can be a simple job that periodically checks for unpublished events and sends them to the message broker.
Following diagram demonstrates the outbox pattern’s behavior. There is a database that is used for storing business related data, and an outbox table. As we see in the diagram, when a service performs an operation like inserting, updating or deleting business related data, this event is also inserted into the outbox table at the same time. Then there is an outbox processor which reads the events from the outbox table and publishes them to the message broker.
If we look at the architectural structure of an outbox pattern, it should contain following components:
- application service logic that supports operations:
- business logic that performs some operations, updates the application state and generates events
- outbox persistence mechanism that stores the events into outbox table
- outbox entity, that stores fields like event id, event type, event payload and some metadata such as timestamps
- outbox processor, which is a background process that periodically scans the outbox table for new events and publishes them to the message broker. This component also needs to take care of the error handling and retries for failed event publications
- message broker that delivers events to the subscribed services
- downstream systems that are consuming and processing events from the message broker
As mentioned, the outbox processor contains logic that scans the outbox table for new events and then publishes them. Once the event is published, we need to make sure to appropriately process this record in the outbox table to avoid being resend again. We can either delete this record, or mark it as sent with a boolean field for example. In case of later, we have to pay attention that the outbox pattern is not meant for the auditory purposes but for the reliability. Therefore send events should be deleted eventually to avoid the outbox table to grow indefinitely.
Another thing that is in the scope of the outbox processor’s task, is to implement error handling and retry mechanisms in case of failed events. Retrial mechanism is important, because it will try to resend the events that failed previously. This guarantees at least once delivery. On the downside, this mechanism can lead to duplicate messages being sent. This can happen if the error occurs after the event is being sent and we fail to update the event. Next time the outbox processor scans for the unpublished events, this event will still be marked as unsent and will be delivered to the message broker again. One solution to mitigate this issue is to implement idempotent consumers. They should expect the same messages can be sent and they should handle this properly.
The Outbox pattern works well with the CDC (Change Data Capture) pattern. Some databases support mechanisms to capture changes on a database level and publish events based on the changes. This solution eliminates the need for an additional table or the scheduler that needs to periodically scan for the unpublished events.
Use cases
Outbox pattern is useful when we want to guarantee atomicity of operations across multiple services. It’s useful when we are dealing with services where data consistency is highly important, such as in the financial industry. The outbox pattern ensures the events are published only in case the operation in the database succeeded and guarantees the events are published in order they were generated. One of the drawbacks of the outbox pattern is that it can cause multiple messages being sent. We can bypass this issue by implementing an idempotent message handler.
References:
- Richardson, Chris. Microservices Patterns. Manning Publications, 2019.
- Kritiotis, Panayiotis. ”Outbox pattern – Why, How and Implementation Challenges”, 2021, https://pkritiotis.io/outbox-pattern-implementation-challenges/
- Ozkaya, Mehmet. “Outbox Pattern for Microservices Architectures”, 2021, https://medium.com/design-microservices-architecture-with-patterns/outbox-pattern-for-microservices-architectures-1b8648dfaa27
- The Java Trail. “Consistency in microservices: transactional outbox pattern”, Stackademic, 2023, https://dip-mazumder.medium.com/consistency-in-microservices-transactional-outbox-pattern-bcd9d3b08676