Runbook — order events lagging
Diagnose and clear a backlog on the OrderPlaced consumer.
SECTION 01 · Note
When to use this
Warning
Use when the OrderPlaced consumer lag alarm fires (lag > 5 min) or downstreams report stale orders.
Procedure
SECTION 02 · Flowchart
Triage the lag
FLOW
1yes2no3yes4no
Commands
Check consumer lagbash
aws cloudwatch get-metric-statistics --namespace Orders \ --metric-name ConsumerLag --period 60 --statistics Maximum
Scale the consumer servicebash
aws ecs update-service --cluster orders --service event-worker --desired-count 6
Symptom → action
| Symptom | Likely cause | Action |
|---|---|---|
| DLQ growing | Poison message | Inspect, fix or skip, then replay the DLQ |
| CPU pinned | Under-provisioned | Scale consumers up; right-size after |
| No throughput | Bus outage | Escalate to platform on-call |