A causal chain from a real Resolve investigation

Report · PostgreSQL replication lag · Thu 8:23 AM

A gateway pagination bug crashed a pod, and its stuck query paused replication on a read replica

  1. 01
    Pagination drops the page tokenpaginatedK8sApiCall passes continue, but the Kubernetes client reads _continue. A unit test asserted the same wrong key.
    code
  2. 02
    Long Kubernetes lists loop on page 1About 10 list calls a minute, every one for page 1, until the pod died.
    logsmetrics
  3. 03
    Those scrapes never finishEvery crashed pod had started one 17 to 19 minutes before it crashed.
    logs
  4. 04
    Uncapped list loading fills the heapAbout 89% of in-use heap came from parsing those lists, collected into one array with no size cap.
    heap profilecode
  5. 05
    A gateway pod runs out of memory and freezesMemory climbed from 3.5 to 11.5 GiB in 20 minutes, then a fatal heap error.
    logsmetrics
  6. 06
    Its read query gets stuck sending rowsThe query on the replica could not be cancelled until the socket broke at 8:22 AM.
    database logs
  7. 07
    The replica pauses applying writer changesLag rose step for step with the stuck session's age, and replay resumed when it died.
    metrics
  8. 08
    Replication-lag alert firesLag peaked at 205 seconds against a 10-second threshold.
    alert
Fix the page token name in the gatewayEscalated with fixUntil it ships, gateway pods crash every 1.5 to 3 hours. Delete any pod stuck sending rows to a replica.