Checkout recovered at 10:17 UTC after 31 minutes of elevated failures; 18.6% of purchase attempts failed, with no evidence of incorrect charges.1
- A payment-provider timeout interacted with an unbounded synchronous retry path.2
- Disabling the retry flag restored checkout before the provider fully recovered.3
- The permanent fix is to move retries off the request path and cap attempts at two.
Impact
| Measure | Result | Confidence |
|---|---|---|
| Incident window | 09:46-10:17 UTC | confirmed from deploy and alert logs4 |
| Failed attempts | 1,284 | confirmed from checkout events1 |
| Affected customers | 973 | deduplicated by customer ID1 |
| Incorrect charges | 0 observed | payment ledger reconciled5 |
Timeline
| UTC | Event | Decision |
|---|---|---|
| 09:46 | Provider latency crosses 4 seconds | Alert fires after error rate exceeds 5% |
| 09:51 | On-call confirms retry amplification | Incident declared SEV-2 |
| 10:02 | Recent application deploy ruled out | Focus shifts to provider and retry path |
| 10:11 | Synchronous retries disabled | Error rate begins falling immediately |
| 10:17 | Checkout returns below 1% failures | Incident moves to monitoring |
Root cause
The provider slowdown was the trigger, not the complete cause. Checkout retried timed-out authorization requests three times inside the original request. Each attempt consumed another worker slot, so a partial provider slowdown became local queue saturation.2
The feature flag was intended to improve recovery from brief network failures. It had no total time budget and was enabled globally without a saturation guard.3
What changes
| Owner | Change | Due | Verification |
|---|---|---|---|
| Payments | Move retries to the background queue | Jul 14 | Load test holds p95 below 2s during injected timeouts |
| Platform | Add provider-specific circuit breaker | Jul 16 | Breaker opens before worker utilization reaches 75% |
| Data | Add failed-checkout recovery cohort | Jul 12 | Support can identify and contact affected customers |
| On-call | Add retry-amplification runbook | Jul 11 | Game day completed by a non-author |
Customer follow-up
Send a recovery email only to customers whose final attempt failed and who did not complete a purchase within 24 hours. Do not message customers who recovered on retry or imply that they were charged.5