ADR-014 — Idempotency keys on the payments API
Clients supply a key; the API replays the first response instead of charging twice.
Status
Accepted — 2026-03-04. Supersedes ADR-009 (retry budgets). Proposed → Accepted → Superseded.
Context
Mobile clients retry POST /payments whenever the network drops, and a retry
that lands after the charge succeeded bills the customer a second time. We saw
nine duplicate charges last quarter, every one of them refunded by hand. The
PSP deduplicates on its own reference, but only within a five-minute window,
which is shorter than our longest client retry backoff.
Decision
Idempotency-Key header on every mutating payments call. The API stores the key with the response for 24 hours and replays that response verbatim for any repeat, so a retry can never produce a second charge.What forced it
The forces behind this decision
How it works
The second call never reaches the PSP
/paymentsOptions considered
Three ways to make a retry safe
- The client defines what "the same request" means, so a deliberate second charge still works
- One header, one lookup — the replay path is trivial to reason about and to test
- Matches Stripe and Adyen, so SDK users need no explanation
- Clients must generate and persist a key across retries
- Adds Redis to the payments write path
- Nothing changes for existing clients
- No new client-side state
- Guesses at intent — two legitimate identical charges silently collapse into one
- Needs a body-hash index on every mutating route, not one header
- No code, no new dependency
- Five-minute window, shorter than our own retry backoff
- Our own writes still double up even when the charge does not
- Couples correctness to a vendor behaviour we do not control
Architecture
Where the key is checked
Consequences
| Consequence | What it means for us | Status |
|---|---|---|
| Retries are safe by default | Clients drop their bespoke retry guards; the SDK sends a UUID per user action | completed |
| A new Redis dependency on the write path | Payments now fails closed if the key store is down — capacity and alerting owned by Platform | in progress |
| Keys expire after 24 hours | A retry the next day charges again. Support needs the runbook line, and we watch the replay-rate metric | todo |
What could go wrong
Risks we are accepting, and what we do about them
Mitigation: Fail closed with 503 + Retry-After; Redis runs multi-AZ with a 60s failover drill each release
Mitigation: Store a request fingerprint beside the key and return 422 when it disagrees
Mitigation: Log the miss rate per client; the header becomes required at v2
Rollout
From merged to required
- 1Ship the middleware, keys optional
A request without a key behaves exactly as it does today, so nothing breaks on deploy.
bashPAYMENTS_IDEMPOTENCY=shadow - 2Send keys from the SDKs
One UUID per user action, persisted with the pending payment so a cold start reuses it.
iOS and Android ship on their own cadence — expect six weeks before the miss rate flattens.
- 3Watch the replay rate
Replays should track the retry rate. A replay rate near zero means keys are being regenerated per attempt.
promqlsum(rate(payments_idempotent_replay_total[5m])) by (client) - 4Make the header required
Once the miss rate is under 1% for two weeks, missing keys get 400 on v2. v1 keeps the old behaviour until it is retired.
ADR
example: ADR-014 — Idempotency keys on the payments API
An architecture decision record — status, context, the decision, options, consequences.
calloutdriverssequenceoptionsblock+3