The batch failed and took eleven jobs with it
Nobody knows yet which downstream systems are now holding bad data, and finding out will take longer than fixing the job that started it.
What actually happens
The failed job is the smallest part of this incident. What it fed is the incident.
A job in an overnight chain fails. That much is visible immediately and the job itself is straightforward to fix.
What is not immediately visible is what depends on it. Some downstream jobs will detect the missing input and stall, which is the good outcome. Others will find partial output present, treat it as complete, and run against it.
That second category is where the damage is. Those jobs produce output that is internally consistent and derived from incomplete input, and they in turn feed things.
Establishing the blast radius means walking the dependency chain and, for each job, determining whether it stalled or ran, and if it ran whether its input was complete. In most estates that chain is documented partially, in more than one place, and not always accurately.
By the time the picture is assembled, the operations team frequently discovers that the highest-impact consequence was not the failed job at all but a report that went out at 07:00 from a system four hops downstream.
Fixing the failed job takes minutes. Working out what it fed, and which of those ran anyway, is the part that takes the morning.
The same night, two ways
The comparison is not about repairing the job. It is about how much of the chain ran on bad input before anyone knew the shape of the problem.
Illustrative, not measured. The times below model a scenario built from the patterns we see in production estates. They are not timings recorded at a named customer. The point is the shape of the clock, not the totals: check it against your own last ten incidents.
Today, blast radius assembled by hand
- 01:30Job fails in the overnight chain. Alert raised on the job.
- 01:30 ↓ 02:10WaitingOn-call fixes the job. Downstream chain continues in the meantime.
- 02:10Question raised: what depended on this. Dependency documentation is partial.
- 02:10 ↓ 05:40WaitingChain walked by hand. Some jobs stalled, some ran on partial input. Distinction established job by job.
- 05:40Eleven affected jobs identified. Three ran on partial input.
- 07:00WaitingA downstream report has already gone out. Reissue required.
~4 hours to a blast radius · and the chain kept running
With the chain held on failure
- 01:30Job fails. Sentinel treats the failure as a chain event rather than a job event.
- 01:31Dependency graph resolved from actual execution history rather than from documentation. Eleven downstream jobs identified.
- 01:32Downstream jobs held before execution. Nothing runs on partial input. Business impact ranked across the affected set.
- 01:34Cause identified on the originating job. Action Ticket raised with the fix, the affected chain and the rollback path.
- 01:36Escalation decision made on evidence: only the two jobs feeding SLA-bound outputs warrant waking anyone.
- 02:05Approved and applied. Chain released in dependency order. Sherlock verifies record counts at each hop.
~35 minutes · nothing downstream ran on partial input
The four hours in the first column is almost entirely blast radius assembly, and the damage during it is caused by the chain continuing while that assembly happens.
Holding the chain is the whole mechanism and it is reversible, which is why it can be done immediately without an approval. Releasing the chain is not reversible in the same way, so that waits.
Resolving the dependency graph from actual execution history rather than from documentation is what makes the hold trustworthy. Documented chains are incomplete in every estate we have seen; observed chains are complete by construction.
Ranking by business impact changes who gets woken. Eleven affected jobs is not eleven escalations, and being able to say that only two feed anything SLA-bound is the difference between a paged team and a morning handover.
Why the number is what it is
The four hours in the first column is almost entirely blast radius assembly, and the damage during it is caused by the chain continuing while that assembly happens.
Holding the chain is the whole mechanism and it is reversible, which is why it can be done immediately without an approval. Releasing the chain is not reversible in the same way, so that waits.
Resolving the dependency graph from actual execution history rather than from documentation is what makes the hold trustworthy. Documented chains are incomplete in every estate we have seen; observed chains are complete by construction.
Ranking by business impact changes who gets woken. Eleven affected jobs is not eleven escalations, and being able to say that only two feed anything SLA-bound is the difference between a paged team and a morning handover.
Resolving the dependency graph from actual execution history rather than from documentation is what makes the hold trustworthy.
Who decides to press go
Holding a chain and releasing a chain are asymmetric actions and are gated asymmetrically.
Holding the downstream chain runs under policy immediately. It is fully reversible, it prevents partial-input processing, and delay is a recoverable cost.
The Action Ticket carries the fix, the affected job list, the record count expectations and the rollback path, and waits for the operations owner to approve before anything is released or reprocessed.
This asymmetry is the same one that appears throughout the platform. Stopping something is safe and happens immediately. Starting something is not, and it waits for a name.
Job fails in an overnight chain. Downstream jobs scheduled to run. Some will stall on missing input, others will proceed on partial output.
Dependency graph resolved from actual execution history. Affected downstream jobs identified and ranked by business impact, including which feed SLA-bound outputs.
Downstream chain held before execution. Action Ticket raised with the fix and affected chain. Escalation limited to the jobs that warrant it.
Partial-output detection added at each hop so a downstream job cannot treat incomplete input as complete. Observed dependency graph retained as the authoritative chain.
This is a platform capability, not a published customer deployment for this exact scenario. The mechanism, which is cross-system correlation followed by governed MOP execution with pre-check, post-check, rollback and approval gating, is running in production today across managed estates; see governed day-2 operations across 2,000+ nodes and closed-loop network automation. The timings shown are modelled, not measured at a named customer.
If the action carries no service impact
Holding the downstream chain runs under policy immediately. It is fully reversible, it prevents partial-input processing, and delay is a recoverable cost.
If it releases the chain or reprocesses data
The Action Ticket carries the fix, the affected job list, the record count expectations and the rollback path, and waits for the operations owner to approve before anything is released or reprocessed.
What Sentinel did, step by step
- ObserveJob fails in an overnight chain. Downstream jobs scheduled to run. Some will stall on missing input, others will proceed on partial output.
- InvestigateDependency graph resolved from actual execution history. Affected downstream jobs identified and ranked by business impact, including which feed SLA-bound outputs.
- ActDownstream chain held before execution. Action Ticket raised with the fix and affected chain. Escalation limited to the jobs that warrant it.
- OptimizePartial-output detection added at each hop so a downstream job cannot treat incomplete input as complete. Observed dependency graph retained as the authoritative chain.
Bring us a batch failure from last month
We will resolve the real dependency chain and show what ran on partial input.