The connection pool ran out and took checkout with it
The database was healthy. The pool in front of it was not. Every symptom the team could see pointed at the database, so that is where the first twenty minutes went.
What actually happens
This incident is not hard to fix. It is hard to look at, because every symptom points confidently in the wrong direction.
Traffic shifts, either from a marketing event or from a retry storm elsewhere, and the connection pool in front of the primary database saturates. Requests queue for a connection and time out waiting.
From the application side the symptoms are database timeouts. From the customer side, checkout fails. From the on-call engineer side, the obvious first hypothesis is that the database is in trouble, and that hypothesis is wrong.
The database is fine. Its CPU is unremarkable, its query latency is normal, its lock waits are nothing special. It is not doing much, because the pool in front of it is not letting anything through.
The team loses time here, and the time is not wasted through incompetence. It is wasted because checking whether the database is healthy is the correct first step, and the answer to that check is misleading unless you also look at the pool in the same breath.
Meanwhile the failure cascades. Services that call checkout begin timing out themselves. Their own pools start to saturate under the retries, and the incident widens from one component to several.
Everything the team could see said database. The database was the one component in the chain that was working normally.
The same hour, two ways
The comparison here is about which component gets looked at first, and what the cascade does while that is being worked out.
Illustrative, not measured. The times below model a scenario built from the patterns we see in production estates. They are not timings recorded at a named customer. The point is the shape of the clock, not the totals: check it against your own last ten incidents.
Today, following the symptoms
- 14:12Traffic shift saturates the connection pool. Checkout begins failing with database timeouts.
- 14:14Alert fires on checkout error rate. On-call engaged.
- 14:14 ↓ 14:33WaitingDatabase investigated first: CPU, query latency, lock waits, slow query log. All normal.
- 14:33Pool metrics checked. Saturation identified.
- 14:33 ↓ 14:45WaitingUpstream services now also degrading under retries. Scope widened.
- 14:58Pool size increased and retry policy capped. Cascade unwinds.
~45 minutes · and the cascade spread during the misdirection
With Sentinel checking both together
- 14:12Pool saturation and checkout error rate observed together. Sentinel opens one investigation.
- 14:14Pool utilisation, database health, query latency, lock waits, upstream retry behaviour and the traffic shift correlated in one pass.
- 14:16Database ruled out on evidence rather than by elimination. Pool identified as the constraint.
- 14:17Cascade risk flagged: two upstream services already increasing retries against a saturated path.
- 14:18Action Ticket raised with the pool increase and the upstream retry cap, blast radius and rollback path. Owning team notified.
- 14:31Approved. Applied. Sherlock confirms checkout recovers and upstream retries subside.
~19 minutes · and the cascade was capped before it widened
The nineteen minutes against forty-five is not the interesting number. The interesting number is what the cascade did in between, because a saturated path under retries recruits its callers into the incident.
Ruling the database out took nineteen minutes in the first column and two in the second, and both teams did the same check. The difference is that one checked the database and the pool together, and the other checked them in sequence because the symptoms suggested an order.
Symptoms are a bad guide to investigation order, and they are the only guide a human under pressure has. Sentinel does not follow the symptom, it queries the whole dependency chain at once, so a component that is healthy is ruled out on evidence rather than after a detour.
Capping the upstream retries is the part that saves the most and gets the least attention. A retry policy that is reasonable under normal conditions is an amplifier under saturation, and catching that in the first few minutes is the difference between one degraded service and five.
Why the number is what it is
The nineteen minutes against forty-five is not the interesting number. The interesting number is what the cascade did in between, because a saturated path under retries recruits its callers into the incident.
Ruling the database out took nineteen minutes in the first column and two in the second, and both teams did the same check. The difference is that one checked the database and the pool together, and the other checked them in sequence because the symptoms suggested an order.
Symptoms are a bad guide to investigation order, and they are the only guide a human under pressure has. Sentinel does not follow the symptom, it queries the whole dependency chain at once, so a component that is healthy is ruled out on evidence rather than after a detour.
Capping the upstream retries is the part that saves the most and gets the least attention. A retry policy that is reasonable under normal conditions is an amplifier under saturation, and catching that in the first few minutes is the difference between one degraded service and five.
Symptoms are a bad guide to investigation order, and they are the only guide a human under pressure has.
Who decides to press go
Increasing a pool size and capping a retry policy both change behaviour on a live customer path, so both are proposed rather than applied.
Investigation and correlation run under policy. Where a pre-approved pool adjustment exists within safe bounds, it executes with pre-check, post-check and armed rollback.
The Action Ticket is raised and held with the proposed changes, the projected effect on the database, the cascade risk and the rollback path attached. The owning team approves.
The cascade risk is included in the ticket deliberately, because it changes the urgency of the approval and the approver should see it rather than discover it.
Checkout error rate rising with database timeout symptoms. Connection pool utilisation at saturation. Traffic shift under way.
Pool utilisation, database health, query latency, lock waits, upstream retry behaviour and traffic pattern correlated in one pass. Database ruled out on evidence. Pool identified as the constraint.
Action Ticket raised with pool increase and upstream retry cap, cascade risk and rollback path. Held for the owning team.
Pool utilisation promoted to a first-class signal alongside database health. Retry amplification added to the saturation model so the cascade is predicted rather than observed.
This is a platform capability, not a published customer deployment for this exact scenario. The mechanism, which is correlated investigation followed by governed MOP execution with pre-check, post-check, rollback and approval gating, is running in production today; see governed day-2 operations across 2,000+ nodes, infrastructure observability across a large server estate and closed-loop network automation. The timings shown are modelled, not measured.
If the action carries no service impact
Investigation and correlation run under policy. Where a pre-approved pool adjustment exists within safe bounds, it executes with pre-check, post-check and armed rollback.
If it changes retry behaviour or exceeds the approved bounds
The Action Ticket is raised and held with the proposed changes, the projected effect on the database, the cascade risk and the rollback path attached. The owning team approves.
What Sentinel did, step by step
- ObserveCheckout error rate rising with database timeout symptoms. Connection pool utilisation at saturation. Traffic shift under way.
- InvestigatePool utilisation, database health, query latency, lock waits, upstream retry behaviour and traffic pattern correlated in one pass. Database ruled out on evidence. Pool identified as the constraint.
- ActAction Ticket raised with pool increase and upstream retry cap, cascade risk and rollback path. Held for the owning team.
- OptimizePool utilisation promoted to a first-class signal alongside database health. Retry amplification added to the saturation model so the cascade is predicted rather than observed.
Bring us an incident that blamed the database
We will show you what the layer in front of it was doing at the same moment.