Seventy-eight percent of the way to a timeout
Nothing has failed. That is exactly the problem, because the only alert configured is the one that fires after the first customer has already been declined.
What actually happens
Most payment monitoring is built to tell you that transactions started timing out. By then the interesting decision has already been made for you.
The timeout budget on a payment authorisation is fixed by the scheme and by the customer experience: past a certain point the transaction fails and the customer sees a decline. Under normal load the p99 latency sits comfortably inside it.
Through the afternoon the p99 climbs. It is not near the limit, then it is at about half, then it is at about three quarters. Nothing has timed out yet. Every dashboard is green, because green is defined as "no timeouts".
The headroom is the thing being consumed, and headroom is not a metric anyone alarms on. It is the difference between two numbers, one of which is a policy constant that lives in a configuration file rather than in the monitoring stack.
The failure, when it comes, is not gradual. Timeouts are a cliff: the latency distribution shifts far enough that the tail crosses the budget, and then a meaningful share of transactions fail at once. The alert that finally fires is a spike in declines, which is the outcome, not the cause.
The teams who handle this well have someone senior who watches the p99 against the budget by habit. That works until that person is on leave, which is a fragile way to run a payment estate.
Alerting on timeouts is alerting on the customer. By the time the metric moves, the decline has already happened to somebody.
The same afternoon, two ways
The comparison here is not fast repair against slow repair. It is repair before the cliff against repair after it.
Illustrative, not measured. The times below model a scenario built from the patterns we see in production estates. They are not timings recorded at a named customer. The point is the shape of the clock, not the totals: check it against your own last ten incidents.
Today, alerting on the outcome
- 14:00p99 latency begins climbing under load. Well inside the timeout budget. Nothing alerts.
- 14:00 ↓ 16:20WaitingHeadroom falls from comfortable to roughly a fifth. Still no timeouts, still no alert.
- 16:20Tail crosses the budget. Transactions begin timing out. Decline spike alert fires.
- 16:20 ↓ 17:05WaitingTriage from the decline spike backwards: which service, which dependency, which resource.
- 17:05Saturated downstream dependency identified. Capacity added.
- 17:20Latency recovers. Declines stop. The window between 16:20 and 17:20 is lost volume.
~60 minutes of declines · alert fired after the first failure
With Sentinel watching headroom
- 15:05Sentinel flags headroom, not latency: p99 has consumed a growing share of the timeout budget on a steady slope.
- 15:07Service latency, downstream dependency saturation, connection pools, queue depth and recent deploys correlated.
- 15:11Probable cause: one downstream dependency approaching saturation under the afternoon load profile.
- 15:12Projection attached: at the current slope, the tail reaches the timeout budget in roughly 70 minutes.
- 15:13Action Ticket raised with the scaling MOP, blast radius and rollback path. Payments engineering notified.
- 15:26Approved. Capacity added ahead of the cliff. Sherlock confirms headroom restored. No transaction timed out.
~20 minutes · zero declines, because the cliff was never reached
This is the clearest case on the site for why detection timing matters more than repair speed. The repair is identical in both columns. The difference is entirely which side of the cliff it happened on.
In the first column the fix arrives after roughly an hour of failed transactions. In the second, the same fix arrives before any transaction fails at all, and the incident record says "prevented" rather than "resolved".
The mechanism is treating headroom as the monitored quantity. Not the latency, and not the timeout count, but the distance between the current tail and the budget, watched as a trend with a projection attached.
The honest caveat: this only works where the budget is actually known to the platform. If the timeout constant lives only in application configuration and is never surfaced, no monitoring system can compute the headroom. Making that number visible is usually a one-line change and it is the highest-leverage thing on this page.
Why the number is what it is
This is the clearest case on the site for why detection timing matters more than repair speed. The repair is identical in both columns. The difference is entirely which side of the cliff it happened on.
In the first column the fix arrives after roughly an hour of failed transactions. In the second, the same fix arrives before any transaction fails at all, and the incident record says "prevented" rather than "resolved".
The mechanism is treating headroom as the monitored quantity. Not the latency, and not the timeout count, but the distance between the current tail and the budget, watched as a trend with a projection attached.
The honest caveat: this only works where the budget is actually known to the platform. If the timeout constant lives only in application configuration and is never surfaced, no monitoring system can compute the headroom. Making that number visible is usually a one-line change and it is the highest-leverage thing on this page.
The mechanism is treating headroom as the monitored quantity.
Who decides to press go
Adding capacity to a live payments path is lower risk than most remediations, but it is still a change to a production system under load.
Where the scaling action is reversible and carries no service impact, the Action Ticket executes under policy: pre-check, execute, post-check, verify, with automatic rollback if headroom does not recover.
The ticket is raised and held with the proposed scaling action, the projected time to cliff, the blast radius and the rollback path. Payments engineering approves before it runs.
The projection is what makes the approval decision easy. The approver is not being asked to trust a diagnosis, they are being shown how long they have.
p99 latency consuming a growing share of the timeout budget on a steady slope. No timeouts yet. No configured threshold crossed.
Service latency, downstream dependency saturation, connection pools, queue depth and recent deploys correlated. One dependency approaching saturation under the afternoon load profile.
Action Ticket raised with a time-to-cliff projection attached. Scaling MOP staged with rollback, held for payments engineering to approve.
Timeout budget surfaced to the platform as a first-class constant. Headroom added as a monitored quantity per service. Afternoon load profile added to the capacity model.
This is a platform capability, not a published customer deployment. The mechanism, which is correlated investigation followed by governed MOP execution with pre-check, post-check, rollback and approval gating, is running in production today in our carrier estates; see closed-loop network automation and carrier-scale observability across 27,000+ devices. The payments scenario above is that same mechanism applied to a transaction estate. The timings shown are modelled, not measured at a named payments provider.
If the action carries no service impact
Where the scaling action is reversible and carries no service impact, the Action Ticket executes under policy: pre-check, execute, post-check, verify, with automatic rollback if headroom does not recover.
If the change carries service impact
The ticket is raised and held with the proposed scaling action, the projected time to cliff, the blast radius and the rollback path. Payments engineering approves before it runs.
What Sentinel did, step by step
- Observep99 latency consuming a growing share of the timeout budget on a steady slope. No timeouts yet. No configured threshold crossed.
- InvestigateService latency, downstream dependency saturation, connection pools, queue depth and recent deploys correlated. One dependency approaching saturation under the afternoon load profile.
- ActAction Ticket raised with a time-to-cliff projection attached. Scaling MOP staged with rollback, held for payments engineering to approve.
- OptimizeTimeout budget surfaced to the platform as a first-class constant. Headroom added as a monitored quantity per service. Afternoon load profile added to the capacity model.
Bring us a decline spike you alerted on
We will show you how much warning was available before it fired.