Major Incident Postmortem & Chronology Incident Resolved

Incident ID: INC-20260908 Severity: P1 Critical Impact Duration: 42 minutes

Time to Detect (TTD)

2m 14s

Time to Mitigate (TTM)

18m 40s

Total Downtime

42m 00s

Data Loss Rate

0.00%

Traffic Restored & Canary Passes All Health Checks

14:42:10 UTC

Failover cluster in us-west fully assumed traffic. Latency returned to 38ms baseline. Incident marked resolved.

View health-check recovery snapshot

[OK] ingress-gw us-west-02 healthy: 48/48 nodes [OK] p99 latency 38ms (baseline), error rate 0.00% [INFO] Incident INC-2026-0908 marked RESOLVED by @sre-oncall

Initiated Automated Traffic Drain to Backup Region

14:18:25 UTC

SRE team executed Runbook #104. Edge CDN redirected 100% of ingress requests to secondary availability zone.

View executed operations log

[ACTION] Cloudflare route drain: applied zone us-east-01 -> weight: 0 [STATUS] 200 OK: Traffic migrated to us-west-02

Database Connection Pool Exhaustion Triggered

14:00:10 UTC

High memory spike caused unindexed analytical query to hold 100% of active pooled connections, blocking HTTP ingress.