Payment Workflows: Idempotency, Ledgers and Reconciliation

Payment workflows: preserve uncertainty until it is resolved

This is a software design exercise for a merchant integrating a payment provider. It does not reproduce Stripe or PayPal's internal architecture. The goal is a traceable payment state machine with controlled retries. An ACID database alone cannot make a database commit and an external card charge one atomic operation.

Define the operation, not just the HTTP request

Suppose an order costs GBP 25.00. Store the currency and the amount in its defined minor unit—2,500 pence in this example—using integer or suitable decimal arithmetic. Currency-specific units and provider rules must be explicit; never assume every currency has two decimal places.

Use separate identifiers for the order, logical payment attempt, provider object, and individual network attempts. A client retry of the same attempt should reuse its operation key. A genuinely new attempt after a confirmed decline should receive a different identity. Bind the key to the authenticated customer and a hash of the amount, currency, and order; reject reuse with different parameters.

Stripe's API documents provider-side idempotency behavior, including retention and which results are saved. This is a concrete API contract to integrate with, not proof of unlimited end-to-end exactly-once execution. Stripe: idempotent requests.

Walk through the ambiguous timeout

1. A transaction inserts payment attempt P as pending, with a unique operation key, and creates a dispatch event.
2. A worker calls the provider using P's stable idempotency key.
3. The provider accepts the request, but its response never reaches the worker.
4. P remains pending or moves to unknown; it is not labeled declined.
5. A retry with the same key, a provider lookup, or a verified event supplies evidence of the outcome.
6. A local transaction records the provider reference and the permitted transition, with unique constraints preventing duplicate financial postings.

Two concurrent requests with the same key compete on the unique record. The loser reads or waits for the existing operation rather than charging again. A database transaction should not stay open while the worker waits on an external API.

If the provider's key retention window expires while the outcome remains unknown, blindly resending can become a new charge. Escalate or reconcile using the persistent provider reference and business operation identity. The safe answer to “did it succeed?” may temporarily be “we are checking.”

Make ledger entries balanced and explainable

The ledger records economic events separately from mutable workflow status. For a simplified settled receipt of GBP 25, one journal could debit a cash or processor-clearing asset by GBP 25 and credit the merchant payable liability by GBP 25. Fees, refunds, disputes, and settlement timing require their own accounts and journals. Debit is not universally “negative”: its effect depends on the account type.

Each journal must balance within one currency, and every entry has a journal ID, account, currency, amount, and direction. Enforce the journal's validity in the same transaction that posts it. Correct a posted error with a linked reversal or adjustment, preserving the history.

A cached balance can be useful and correct when updated transactionally or derived from a known ledger position. The problem is an unexplained mutable balance with no auditable history, not the mere existence of a balance column.

A late webhook is normal, not a second payment

A browser redirect is not authoritative evidence that money moved. Treat provider events as signed external input, store the event ID uniquely, and make processing repeatable. Events may be retried and may arrive out of order. Stripe documents these properties and signature verification requirements. Stripe webhook documentation.

For example, receiving the same success event three times should create one transition and one journal. A later arrival for an older pending state must not move a settled payment backward. If the event lacks enough context, retrieve the current provider object before deciding what to do.

Reconciliation compares evidence, not just rows

Reconciliation joins local operations, provider transactions, settlements, fees, refunds, and disputes using stable references and currency. A payment absent from today's settlement file is not automatically failed; it may settle later, be authorized but uncaptured, or belong to another reporting interval.

Place discrepancies into reasoned categories: missing local event, amount mismatch, timing difference, duplicate posting, or unresolved external outcome. Automate a correction only when the evidence justifies it. Preserve who or what resolved the discrepancy.

A practical drill sends 20 concurrent retries for P, drops the first provider response, replays the success event, and restarts the ledger worker between writes. Check that the operation has one provider outcome, its journal balances, no duplicate posting exists, and unresolved states remain visible. Availability, recovery, auditability, and security all matter alongside consistency; “CP” is not a complete payment design.