Deploying Data Pipelines using the Saga pattern

Shibi Ramachandran 9 Feb, 2023 17 - 4 min read

Question is: how do we ensure the atomicity of the pipeline deployment? We’ve decided to implement the Saga pattern to solve this issue!

What is the Saga pattern?

How does it work?

  1. Orchestration: the process of coordinating the actions of multiple components in order to achieve a larger goal.
  2. Compensation: the process of rolling back any changes that have been made during a transaction if something goes wrong.

How did we implement the Saga pattern?


  1. The PipelineProcessor validates and processes the incoming request and submits it to the Orchestrator and finally communicates the result to the client.
  2. The Orchestrator is the core component that coordinates all the transactions across all the components. It takes a predefined set of pipeline steps and executes them in the given order.
  3. The Orchestrator receives the request and begins executing the transaction by sending a request to the first step in the workflow.
  4. The first step processes the request and sends a response to the Orchestrator in a Kafka topic and the Orchestrator listens to these events.
  5. The Orchestrator acts on the event and executes the next step if it’s a “SUCCESSFUL” event and if it’s a “FAILED” event, a “ROLLBACK” is triggered, resulting in a cascading rollback where all the executed pipeline steps will be rolled back.
  6. If all the steps in the workflow execute successfully, the Orchestrator sends a final response to the client to indicate that the transaction completed successfully.
  7. If one of the steps fails, the Orchestrator executes the compensating transactions (rollback) to undo the changes made by the previous steps, and sends a failed response to the client.
  8. There are some cases where the rollback could also fail. If a rollback fails, a “ROLLBACK_FAILED” status is sent to the client.
  9. When the rollback also fails, the engineers are alerted via slack channels for a manual recovery or cleanup.

Wrapping up:

Want to join Shibi Ramachandran in finding solutions to interesting problems?