DOCUMENTRUNBOOK · ON-CALL

Runbook — order events lagging

Diagnose and clear a backlog on the OrderPlaced consumer.

Note

01 · Note

SECTION 01 · Note

When to use this

Warning

Use when the OrderPlaced consumer lag alarm fires (lag > 5 min) or downstreams report stale orders.

Runbook — order events lagging2 / 5

Procedure

02 · Flowchart

SECTION 02 · Flowchart

FLOW

1yes2no3yes4no

Runbook — order events lagging3 / 5

Commands

03 · Code

SECTION 03 · Code

Check consumer lagbash

aws cloudwatch get-metric-statistics --namespace Orders \
  --metric-name ConsumerLag --period 60 --statistics Maximum

Scale the consumer servicebash

aws ecs update-service --cluster orders --service event-worker --desired-count 6

Runbook — order events lagging4 / 5

Symptom → action

04 · Comparison

SECTION 04 · Comparison

Symptom	Likely cause	Action
DLQ growing	Poison message	Inspect, fix or skip, then replay the DLQ
CPU pinned	Under-provisioned	Scale consumers up; right-size after
No throughput	Bus outage	Escalate to platform on-call

Runbook — order events lagging5 / 5