Something is slow. Nobody can say which something.
Four services blamed each other while the customer waited, and every team was looking at a dashboard that showed them innocent.
What actually happens
Every team in this incident is telling the truth. That is precisely why it takes so long.
End-to-end latency on a customer-facing path rises noticeably. The path crosses four services owned by three teams, plus a shared data layer that nobody quite owns.
Each team opens its own dashboard. Each dashboard shows that service performing within its own normal range, because each service is measuring the time it spends, not the time the request spends waiting for its dependencies.
So the incident call becomes a negotiation. Everyone is technically correct about their own hop, nobody has visibility into the whole path, and the discussion converges on whoever has the weakest evidence rather than on whoever has the problem.
The actual cause is usually a hop that is slightly slower under a particular condition, amplified by a retry policy somewhere upstream. That combination is invisible from any single service view and obvious from the whole path.
Meanwhile the customer-facing symptom continues, and the elapsed time is being consumed by an organisational problem rather than a technical one.
Nobody is hiding anything. Four teams each hold one quarter of the evidence, and no forum exists in which those quarters are assembled faster than by argument.
The same incident, two ways
The clock below measures something unusual: how long it takes an organisation to work out which of its own components is responsible.
Illustrative, not measured. The times below model a scenario built from the patterns we see in production estates. They are not timings recorded at a named customer. The point is the shape of the clock, not the totals: check it against your own last ten incidents.
Today, four dashboards and a call
- 11:05End-to-end latency rises on a customer-facing path. Alert fires on the edge service.
- 11:05 ↓ 11:20WaitingIncident call convened. Three teams join. Each opens its own view.
- 11:20 ↓ 12:10WaitingEach service demonstrates it is within its own normal range. Ownership of the shared data layer debated.
- 12:10Someone manually stitches traces across the full path.
- 12:25Cause identified: one hop slower under a specific condition, amplified by an upstream retry policy.
- 12:40Retry policy adjusted. Latency recovers.
~95 minutes · of which most was establishing who to look at
With Sentinel holding the whole path
- 11:05Latency rises. Sentinel investigates the request path, not the service that alerted.
- 11:07Traces stitched across all four services and the shared data layer. Time attributed per hop including wait time.
- 11:10One hop identified as carrying the excess. Its own service metrics are normal, its dependency wait is not.
- 11:12Upstream retry policy identified as the amplifier. Recent configuration changes on both correlated.
- 11:13Action Ticket raised with the retry policy change, blast radius and rollback path. Owning team notified with evidence attached.
- 11:28Approved. Policy adjusted. Sherlock confirms end-to-end latency returns to baseline.
~23 minutes · and no team had to prove its innocence
The technical fix here is a configuration change that takes a minute. Almost all of the ninety-five minutes in the first column is organisational: convening, arguing, and eventually doing by hand the trace stitching that resolves it.
The reason this pattern is so durable is that it is nobody fault and therefore nobody responsibility. Each team is correctly instrumented for its own service. The gap is between services, and gaps between services do not have owners.
Sentinel treats the request path as the object of investigation rather than the service that raised the alert. Time is attributed per hop including time spent waiting on dependencies, which is the measurement that makes a slow dependency visible from the caller side.
The organisational benefit is larger than the minutes suggest. An incident that arrives with per-hop attribution attached does not need a negotiation, which means the owning team is engaged directly rather than everyone being engaged defensively.
Why the number is what it is
The technical fix here is a configuration change that takes a minute. Almost all of the ninety-five minutes in the first column is organisational: convening, arguing, and eventually doing by hand the trace stitching that resolves it.
The reason this pattern is so durable is that it is nobody fault and therefore nobody responsibility. Each team is correctly instrumented for its own service. The gap is between services, and gaps between services do not have owners.
Sentinel treats the request path as the object of investigation rather than the service that raised the alert. Time is attributed per hop including time spent waiting on dependencies, which is the measurement that makes a slow dependency visible from the caller side.
The organisational benefit is larger than the minutes suggest. An incident that arrives with per-hop attribution attached does not need a negotiation, which means the owning team is engaged directly rather than everyone being engaged defensively.
Sentinel treats the request path as the object of investigation rather than the service that raised the alert.
Who decides to press go
Changing a retry policy on a live path affects load on every downstream service, so it is proposed rather than applied.
Investigation, trace stitching and per-hop attribution all run under policy. None of it changes a running system.
The Action Ticket is raised and held with the proposed change, the projected load effect on downstream services and the rollback path attached. The owning team approves before anything is applied.
The approval goes to the team that owns the hop, not to the team that received the alert. That routing is only possible because the attribution was done first.
End-to-end latency on a customer-facing path rising. Edge service alerts. Four services and a shared data layer in the path, three owning teams.
Traces stitched across the full path. Time attributed per hop including dependency wait. One hop carries the excess; an upstream retry policy amplifies it.
Action Ticket raised with the retry policy change, projected downstream load effect and rollback path. Routed to the owning team, held for approval.
Per-hop attribution retained as a standing view for this path. Retry policy interaction added to the pre-deploy checks for both services.
This is a platform capability, not a published customer deployment for this exact scenario. The mechanism, which is correlated investigation followed by governed MOP execution with pre-check, post-check, rollback and approval gating, is running in production today; see governed day-2 operations across 2,000+ nodes, infrastructure observability across a large server estate and closed-loop network automation. The timings shown are modelled, not measured.
If the action carries no service impact
Investigation, trace stitching and per-hop attribution all run under policy. None of it changes a running system.
If it changes configuration on a live path
The Action Ticket is raised and held with the proposed change, the projected load effect on downstream services and the rollback path attached. The owning team approves before anything is applied.
What Sentinel did, step by step
- ObserveEnd-to-end latency on a customer-facing path rising. Edge service alerts. Four services and a shared data layer in the path, three owning teams.
- InvestigateTraces stitched across the full path. Time attributed per hop including dependency wait. One hop carries the excess; an upstream retry policy amplifies it.
- ActAction Ticket raised with the retry policy change, projected downstream load effect and rollback path. Routed to the owning team, held for approval.
- OptimizePer-hop attribution retained as a standing view for this path. Retry policy interaction added to the pre-deploy checks for both services.
Bring us an incident that needed three teams
We will show how much of that call was attribution rather than diagnosis.