Checks at the end, not at each hop
Quality tests, where they exist, run against the final table. By then the defect has passed through four transformations and finding which one introduced it is archaeology.
From Broken Dashboards to Trusted Data
Reference Architecture
How we build observability for data pipelines and warehouses. The way bad data actually gets discovered today, the quality rules we write instead of the ones we find, the procedures a data incident routes to, and why we measure trust rather than uptime.
The initial state we keep meeting
Ask a data team how they found out about their last serious data incident and the honest answer is usually a message from someone in finance. Not an alert. A person, looking at a number that was obviously wrong, on a Tuesday, several hours after the pipeline that produced it had already reported success.
That is the defining property of the accumulated state, and it is worth naming precisely: the pipeline is green and the data is wrong. Orchestrators report on whether tasks completed. They do not report on whether the output means anything. A job that reads an empty source file, transforms nothing and writes zero rows exits with status success.
Six problems tend to show up together underneath that:
Green pipelines, wrong numbers
The orchestrator asserts that tasks completed. Nothing asserts that the output is plausible. A zero-row load and a correct load look identical in the run history.
No lineage, so no blast radius
When a table is found to be bad, nobody can say quickly which dashboards, models and downstream tables consumed it, so either everything is declared suspect or nothing is.
One failure, forty alerts
An upstream job fails and every descendant task fails behind it. Forty notifications describe one root cause, and the one that names the root cause is not distinguishable from the other thirty-nine.
Thresholds set once, never revisited
Row count tolerances and freshness windows are written when a pipeline is built and never tuned. They fire on every seasonal change and get muted within a quarter.
Trust is not measured at all
Teams track pipeline success rate, which stays near 100 percent, while the analysts they serve quietly stop believing the numbers. Nothing in the reporting captures that.
Why we measure trust and not uptime
The most useful thing we have read on this came from a data team who chose their headline metric to be how much their analysts trusted the data, surveyed and tracked over time, rather than any figure their platform produced. Their point was that trust is a human response and does not move in step with your uptime chart.
We think that is correct and we would go one step further. Pipeline success rate is not merely an incomplete measure of data quality, it is an actively misleading one, because it goes up when you add retries and stays up during precisely the failure mode that matters most. A team optimising it will be congratulating itself in the same quarter its consumers quietly build a shadow spreadsheet.
This is the same argument we make about incident closure elsewhere on this site. A procedure that finished is not a problem that ended, and a pipeline that ran is not a number anyone believes. So the design goal is not fewer failed jobs. It is a shorter distance between the moment data becomes wrong and the moment the people who depend on it know it, with the correction and the affected reports named.
Everything below follows from that goal, and the order it is built in follows from it too.
The solution
Five layers. Each has one job, each depends only on the layer beneath it, and each is worth switching on before the next one exists.
Phase 1: instrument the pipeline, not the dashboard
Checks move from the end of the pipeline to each hop of it. At ingest we assert what arrived. After each transform we assert what the transform should have preserved or produced. At publish we assert what the consumer is entitled to expect. The point is not more tests. It is that a defect gets caught at the hop that introduced it, so the diagnosis is already done by the time anyone is told.
Lineage is captured while the job runs. Column-level lineage read from query plans and job definitions as they execute, not maintained by hand in a catalogue that goes stale in a fortnight. This is the single most important artefact in the whole architecture, because everything above it depends on being able to answer two questions instantly: what fed this table, and what does this table feed.
Profiles are baselined before any rule is written. For several weeks we record what normal looks like per table and per column: arrival times and their spread, row counts by weekday, null rates, cardinality, value distributions. Rules written against a measured baseline are worth having. Rules written against a number somebody guessed at design time are the reason your team mutes alerts.
Nothing is switched off. Existing tests, existing orchestrator alerts, existing dashboards all keep running. The checks land beside them. If somebody proposes ripping out the current test suite in week one, including us, push back.
What exists at the end of phase 1: a lineage graph of the estate, a measured profile per table, and check results recorded at every hop, all held beside the operational telemetry rather than in a separate data-team silo. That last part matters more than it sounds. Data incidents are very often infrastructure incidents wearing a different hat, and a warehouse that slowed down because a node was under memory pressure is one query away from being explained rather than one team away.
Phase 2: rewriting the quality rules
Once there is a baseline and a lineage graph, the rules get rewritten. The middle column is what we typically find in place. The right column is what we write instead.
| Check | The rule we usually find | The rule we write instead |
|---|---|---|
| Freshness | Table has not updated in 24 hours. | Arrival is outside this table's own measured inter-arrival distribution and the upstream job reported completion and today is a normal loading day for it. A table that always lands at 02:10 is late at 03:00, not at 26 hours. |
| Row count | Row count dropped by more than 10 percent. | Row count outside the band measured for this weekday and the source extract did not move by the same proportion. If the source genuinely shrank, that is an upstream business change, not a pipeline defect, and it should not page the data engineer. |
| Schema change | Any schema change on a tracked table. | A column was removed, renamed or narrowed in type and lineage shows at least one downstream consumer of that column. Additive columns with no consumer are recorded, not raised. |
| Null rate | Null rate above a fixed percentage. | Null rate above the column's own rolling baseline and the column participates in a downstream join, filter or metric definition. A sparsely populated free-text field is not an incident. |
| Distribution drift | Mean or median moved by more than a set amount. | Distribution shifted beyond the measured baseline and row count is inside its normal band, so a partial load is reported as a partial load rather than misdiagnosed as a drift. |
| Job failure | Any DAG task failure pages the on-call. | Task failed and the retry budget is exhausted and no upstream dependency in the lineage graph has already failed. One root failure produces one incident, with the descendants attached as consequence rather than as forty separate pages. |
| Reconciliation | Not checked at all in most estates. | Totals in the warehouse agree with totals in the system of record within tolerance, checked on a schedule. This is the check that catches the failures the other six cannot see, and it is almost always the missing one. |
Illustrative rule shapes, not shipped defaults. Every threshold and window here is derived from your own measured baseline.
Three properties make those rules writable at all.
Severity comes from lineage, not from the check. A null rate breach on a column that feeds the regulatory report and the same breach on a column nothing consumes are not the same event, and no severity field set by hand will keep track of which is which as the estate changes. Reading it from the lineage graph means it stays correct without anyone maintaining it.
An operator writes the rules, not a specialist. The rule and formula builder expresses a condition over any check result, run record or profile baseline in the store without writing bespoke test code per table. A rule that spans a check result and an infrastructure metric is one rule here.
Correlation happens before anyone is told. Failures sharing a root task, a source system or a time window group into one incident with one owner. The individual check results still exist and are still queryable. They are not forty notifications.
Phase 3: the procedure the data incident routes to
A correlated data incident with a cited root hop and a named list of affected reports is still a description. Phase 3 gives it an action.
Every step is bracketed the same way it is on the infrastructure side. A pre-check asserts that the conditions the step assumes are still true, which for data work usually means asserting that the upstream fix actually landed before spending an hour of warehouse compute re-running against the same broken input. A post-check asserts that the expectations which failed now pass. Rollback is armed before the step runs. A step whose post-check fails does not continue.
The gate is the blast radius of the action, and in a data estate blast radius reads off the lineage graph. Quarantining one partition of a table nothing consumes yet is a different decision from re-running a job that rewrites the table behind the regulatory report, even when the reasoning behind both is identical.
These are the data-side procedures in our shipped library, matched to the condition that routes to them. The library holds 23 procedures in total and these are the ones that touch data work.
| Correlated condition | Procedure | What it does | Radius |
|---|---|---|---|
| Partition fails expectations | sentinel-dq-quarantine | Quarantine the offending partition, then re-validate expectations against what remains. | One partition. Consumers notified. |
| Upstream fix landed, output still stale | sentinel-pipeline-rerun | Re-run the failed DAG or stage from the given start date and confirm success. | Recompute cost, and rewrites the table. |
| Pipeline config differs from Git | sentinel-config-drift-reconcile | Sync the definition back to its Git desired state through ArgoCD. | Whatever the diff contains. Human gate. |
| Payroll figures disagree across systems | sentinel-cross-system-payroll-reconcile | Orchestrate the coordinated fix and reconciliation across SAP, Workday and ServiceNow. | Cross-system. Human gate, always. |
| Pay calculation ran on bad upstream data | sentinel-workday-payroll-recalc | Re-run the pay calculation after the upstream fix, before disbursement. | Pre-disbursement only. Human gate. |
| Invoice blocked on a three-way match | sentinel-sap-invoice-fix | Post the corrected invoice once the three-way match is resolved. | Financial posting. Human gate. |
How we would measure whether it worked
This is the section where a case study would put results. We do not have a data observability deployment to report, so here is what we would instrument instead, which is the part most write-ups skip.
Who found it first. The single most honest metric in this domain. Of the data incidents in a quarter, what fraction were detected by the platform and what fraction were reported by a human consumer who noticed a wrong number. Capture it for a quarter before anything changes. If that ratio does not move, nothing else you measure matters, and if it does move, almost everything else follows.
Time to detect and time to trust, kept separate. Time to detect runs from the moment the data became wrong to the moment somebody knew. Time to trust runs from the same start to the moment the affected consumers were told it was fixed and went back to using the report. The second is always longer, it is the one your stakeholders experience, and it is the one nobody reports.
Ask the consumers, with the same question every time. A short quarterly survey of the analysts and business users who depend on the data, worded identically each round so the trend means something. This is a soft metric that behaves like a hard one, because it moves late and it moves honestly. Pipeline success rate will tell you things are fine during exactly the period your users stop believing you.
Blast radius accuracy. When an incident names its affected downstream reports, how often was that list right. A confidently wrong list is worse than no list, because people act on it. This is the check on your lineage graph, and it is the one that catches lineage rot before it embarrasses you.
Be careful what you optimise. The number of checks configured is a vanity metric and rises fastest in estates with the worst signal-to-noise. Detection count on its own is similar. Watch the who-found-it-first ratio, and watch how many raised incidents were closed as expected behaviour, which is your false positive rate wearing a polite name.
What this does not solve
It does not tell you what the data should say. The architecture detects that a number moved in a way it has not moved before. Whether the new number is correct is a business question and often only the person who owns the process can answer it. Reconciliation against a system of record is the closest we get, and it only covers what a system of record exists for.
Lineage rot is real. Lineage read from query plans at runtime is accurate for what runs through the platform. Anything that moves data outside it, a manual export, a spreadsheet emailed between two teams, a script on somebody's laptop, is invisible, and the blast radius calculation will be confidently short. Every estate has some of this and it is worth knowing where.
Bad models produce bad data through a healthy pipeline. If a transformation implements the wrong business logic, every check passes and the output is consistently, reliably wrong. Data observability catches deviation from normal, and this failure mode is normal from the first day. It needs review, not monitoring.
Consolidation is a migration, not a switch. Existing tests, dashboards, ownership and habits are built around what you have. The parallel-run phases exist because that does not go away by being ignored.
Looking ahead
The direction this points is a data platform where the question "can I trust this number" has an answer attached to the number itself, visible in the tool the person is already using, rather than being a favour asked of a data engineer in a channel. That is a smaller ambition than most vendors state and a considerably harder one to actually reach, because it is not achieved by adding checks. It is achieved by making the failure path shorter than the human path, and then measuring whether it stayed shorter.
Frequently asked questions
A reference design. No number on this page is a measured result. Our measured deployments are published as case studies with a named scope and a stated basis of measurement for every figure, and none of the current set is a data observability deployment. We would rather label this correctly than borrow numbers from a different engagement.
It is not a replacement for them, it consumes them. Those frameworks are good at expressing an assertion about a table. What they do not do is baseline what normal looks like per table before you write the threshold, scope severity by who is downstream, group a root failure with its forty descendants into one incident, or route the result to a procedure that fixes it and verifies the fix. Keep the tests. The architecture is the layer that makes their output actionable.
Because a large share of data incidents are infrastructure incidents in disguise. A warehouse that slowed under memory pressure, a network path that dropped an extract, a certificate that expired on a source connection. If the run history and check results sit in the same store as the infrastructure signals, that diagnosis is one query. If they sit in a separate catalogue owned by a separate team, it is a meeting.
Not today. Every procedure in the current library carries an approval requirement. Autonomy is granted per action, on the evidence of running that action with approval first, and it is gated on blast radius rather than on the confidence of the diagnosis. On the data side we would expect quarantine to qualify early and anything rewriting a table behind a financial report never to qualify. The governance model is set out in what an action ticket is.
Pick the five tables that appear behind your most-viewed reports. Baseline their profiles for three weeks with nothing alerting. Capture lineage for whatever feeds them. Then turn on freshness and volume rules written against the measured baseline, and nothing else. That is enough to find out whether the who-found-it-first ratio moves, and it costs three weeks rather than a programme.
Yes. The checks, the metadata store, the alert engine and the execution engine all run inside the customer perimeter, and the reasoning layer can run on customer infrastructure with no egress. The tradeoffs of that deployment model are on air-gapped AIOps.
Related reading: DataOps for the product itself, Three Tools, One Signal Store for the same treatment applied to Kubernetes, and how to monitor data pipelines for the practitioner version.