Distributed Transactions

Distributed Transactions: The Consistency Challenge

In a monolith with a single database, transactions are straightforward: BEGIN TRANSACTION → Update User → Update Order → COMMIT. If anything fails, you ROLLBACK, and the database guarantees atomicity.

In microservices, where each service owns its database, you face a fundamental challenge: How do you maintain data consistency across multiple independent databases when a business transaction spans multiple services?

This is the distributed transaction problem.

The Problem Statement

Imagine an e-commerce order flow:

1. Order Service creates an order
2. Payment Service charges the customer
3. Inventory Service reserves items
4. Shipping Service schedules delivery

If Payment succeeds but Inventory fails (out of stock), you've charged the customer for items you can't deliver. You need a way to ensure all services succeed together or all fail together.

A local database transaction cannot atomically cover an unrelated service call. Distributed databases or participants supporting a coordination protocol can provide cross-node transactions, with additional failure and latency trade-offs.

Solution 1: Two-Phase Commit (2PC)

Two-Phase Commit is a distributed algorithm that extends ACID semantics across multiple databases.

How It Works

Phase 1: Prepare (Voting)

1. Coordinator sends PREPARE to all participants
2. Each participant performs the operation but doesn't commit
3. Each participant locks the affected rows/resources
4. Each participant responds: VOTE_COMMIT (ready) or VOTE_ABORT (can't do it)

Phase 2: Commit/Abort (Decision)

1. If ALL voted COMMIT: Coordinator sends GLOBAL_COMMIT to everyone
2. If ANY voted ABORT: Coordinator sends GLOBAL_ABORT to everyone
3. Each participant commits or rolls back accordingly
4. Participants release locks

Sequence Diagram

The Fatal Flaw: Blocking

Scenario: Coordinator crashes after Phase 1 but before Phase 2.

Result: All participants are stuck holding locks indefinitely. They don't know if they should commit or abort. This is called the blocking problem.

Impact:

• Database locks cause contention and deadlocks
• Services become unavailable
• Affected resources can remain blocked; unrelated work may continue

When to Use 2PC

✅ Use when:

• Strict consistency is absolutely required (banking, financial transactions)
• Transactions are short-lived and participants support the required protocol
• Network is reliable and low-latency (same data center)

❌ Avoid when:

• Cross-internet transactions
• Long-running business processes
• High-availability is critical

Real-World Usage

• MySQL XA Transactions: Distributed transactions across multiple MySQL databases
• Java JTA: Enterprise Java beans with distributed transactions
• Operational requirement: Plan recovery of prepared transactions and the coordinator; do not leave in-doubt transactions unmanaged.

Solution 2: SAGA Pattern

Instead of one atomic distributed transaction, SAGA uses a sequence of local transactions, with compensating transactions for rollback.

Two SAGA Approaches

1. Choreography (Event-Driven)

Services publish events, and other services react:

Failure Scenario:

Pros:

• Decoupled services
• No single point of failure
• Highly scalable

Cons:

• Hard to understand the overall flow
• Difficult to debug and monitor
• Cyclic dependencies possible

2. Orchestration (Centralized)

A Saga Orchestrator coordinates the entire flow:

class OrderSagaOrchestrator:
    def execute_order(self, order_id):
        try:
            # Step 1: Create Order
            order = order_service.create_order(order_id)
            
            # Step 2: Charge Payment
            payment = payment_service.charge(order.customer_id, order.total)
            
            # Step 3: Reserve Inventory
            inventory = inventory_service.reserve(order.items)
            
            # Step 4: Schedule Shipping
            shipping = shipping_service.schedule(order.id)
            
            return {"status": "SUCCESS"}
            
        except PaymentFailedException as e:
            # Payment failed, rollback order
            order_service.cancel_order(order_id)
            raise
            
        except InventoryFailedException as e:
            # Inventory failed, rollback payment and order
            payment_service.refund(payment.id)
            order_service.cancel_order(order_id)
            raise
            
        except ShippingFailedException as e:
            # Shipping failed, rollback everything
            inventory_service.release(inventory.id)
            payment_service.refund(payment.id)
            order_service.cancel_order(order_id)
            raise

Pros:

• Explicit business logic in one place
• Easy to understand and debug
• Centralized monitoring

Cons:

• Orchestrator is a single point of failure
• Orchestrator can become a bottleneck
• Tight coupling to orchestrator

Compensating Transactions

Critical concept: You can't "undo" a committed transaction. Instead, you run a new transaction that reverses the effect.

Example:

• Forward Transaction: INSERT INTO orders (id, status) VALUES (123, 'PENDING')
• Compensating Transaction: UPDATE orders SET status = 'CANCELLED' WHERE id = 123

Challenges:

1. Non-Idempotent Operations: Sending an email can't be "unsent"
2. Semantic Compensation: Refunding money isn't identical to "not charging" (audit trail, fees, timing)
3. Partial Compensation: What if the compensation itself fails?

Idempotency is Critical

Services must be idempotent (calling twice has same effect as calling once):

persist a stable operation key before the external call
send the same key and same parameters on every retry
on timeout, query/reconcile the existing operation
only compensate once its outcome is known

This is an application contract, not a payment provider API. Use the provider's documented idempotency mechanism and retention window; changing a key on retry can cause a second charge.

Solution 3: Outbox Pattern

Ensures transactional messaging: Database write + Event publish must be atomic.

The Problem

# NOT ATOMIC!
def create_order(order):
    db.insert_order(order)  # Step 1: DB write
    kafka.publish("OrderCreated", order)  # Step 2: Event publish
    # What if Kafka fails? Order is created but no event published!

The Solution: Outbox Table

BEGIN TRANSACTION;
  INSERT INTO orders (id, customer_id, total) VALUES (123, 456, 99.99);
  INSERT INTO outbox (event_type, payload) VALUES ('OrderCreated', '{"order_id": 123}');
COMMIT;

A background worker reads the outbox table and publishes events. This sketch omits worker locking, durable broker acknowledgements, and retry limits. If it crashes after publication but before marking the row, it can publish twice; consumers must deduplicate by event ID:

while True:
    events = db.execute("SELECT * FROM outbox WHERE published = false LIMIT 100")
    for event in events:
        kafka.publish(event.event_type, event.payload)
        db.execute("UPDATE outbox SET published = true WHERE id = ?", event.id)
    time.sleep(1)

Benefits:

• Atomic: Database write and event "intent" in same transaction
• Recoverable: Publication intent survives a crash; eventual delivery still requires worker recovery, retries, and a functioning broker.
• Polling: Worker can retry on Kafka failures

Comparison Table

Real-World Examples

Illustrative food-delivery workflow

SAGA Orchestration:

1. Place Order (Order Service)
2. Authorize Payment (Payment Service)
3. Notify Restaurant (Restaurant Service)
4. Assign Driver (Dispatch Service)

Compensations:

• If restaurant rejects → Refund payment, cancel order
• If no driver available → Cancel restaurant, refund payment, cancel order

Illustrative inventory workflow

Outbox Pattern:

• Product purchase → Local DB transaction + Outbox record
• Background worker → Publishes InventoryReduced event
• Warehouse Service → Consumes event and updates physical inventory

Interview Tips 💡

When discussing distributed transactions in system design interviews:

1. Start with the problem: "We have Order Service and Payment Service with separate databases..."
2. Choose the boundary: Explain whether atomic cross-resource commit is required or business compensation is acceptable.
3. Explain SAGA: Walk through a concrete example (e-commerce order)
4. Mention trade-offs: "We'll have eventual consistency, so users might see 'Processing' status briefly"
5. Discuss idempotency: "Each service must handle duplicate requests gracefully"
6. Bring up Outbox: "Persist the event intent with the state change, retry publication, and deduplicate consumers."

Common Pitfalls

⚠️ Timeout Confusion: If a service times out, did it succeed or fail? Use idempotent operations and unique request IDs.

⚠️ Compensation Order: Reverse the order of compensations (LIFO). If you booked flight → hotel → car, cancel car → hotel → flight.

⚠️ Testing: Test failure scenarios! Use chaos engineering to randomly fail services and verify compensations work.

Related Concepts

• ACID vs BASE — Consistency models in databases
• Event Sourcing & CQRS — Alternative to distributed transactions
• Message Queues — Reliable event delivery
• Microservices — Service architecture patterns
• CAP Theorem — Fundamental trade-offs in distributed systems

Sources and a failure test

PostgreSQL prepared transactions explains an actual 2PC participant. AWS transactional outbox explains the local commit/publication boundary. Test a timeout after an external operation succeeds and a crash between outbox delivery and acknowledgement; neither failure should be treated as proof that the operation never happened.