Playbook
How to Monitor Data Pipelines (Data Ops)
Written by
Dilip Namdev
Published
Reading time
2 min read
Monitoring data pipelines means watching not just whether jobs run but whether the data they produce is fresh, complete and correct, so a silent data failure is caught before it reaches dashboards and decisions.
A data pipeline can succeed technically while producing wrong data, and that silent failure is worse than a crash because nobody notices until a report is off. Data Ops monitoring watches four dimensions, freshness, volume, schema and lineage, so a broken pipeline is caught by its output, not just its exit code.
Job success is not data success
A job that completes with exit code zero tells you it ran, not that the data is right. It can process a truncated source, drop rows, or write stale data and still 'succeed'. Effective monitoring watches the data itself: is it fresh, is the row count in the expected range, does the schema still match, did upstream sources arrive.
The four signals of pipeline health
Freshness (is the data up to date), volume (is the row or record count in the normal range), schema (has the structure changed unexpectedly), and lineage (did all upstream inputs arrive and are they healthy). Together these catch the silent failures that a job-status check alone misses, and tie a downstream data problem to the pipeline stage that caused it.
- Monitor freshness. Track when each dataset was last updated against its expected schedule.
- Monitor volume. Alert when row or record counts fall outside the normal range.
- Monitor schema. Detect unexpected structural changes that break downstream consumers.
- Track lineage. Confirm upstream inputs arrived and are healthy before trusting the output.
- Act on data incidents. Retry, reroute or hold the pipeline, and alert with the failing stage identified.
How Opstral monitors data pipelines
Opstral's Data Ops pillar treats freshness, volume, schema and lineage as first-class signals, and Sentinel AI acts on data incidents, a failed batch, a stalled stream, a freshness breach, through governed Action Tickets, so a silent data failure becomes an incident that gets resolved.
Frequently asked questions
How is data pipeline monitoring different from job monitoring?
Job monitoring checks whether a pipeline ran; data pipeline monitoring checks whether the data it produced is fresh, complete and correct. A job can succeed while the data is wrong.
What should I monitor in a data pipeline?
Freshness, volume, schema and lineage, so you catch silent data failures and can tie a downstream problem to the stage that caused it.
How do we handle pipelines that legitimately vary a lot in volume?
Baseline against seasonally-adjusted expectations rather than a fixed threshold. If Monday morning always has 3x the volume of Sunday night, that pattern is the baseline; alerting on 'volume higher than Sunday night' would fire every Monday. Modern data observability tools handle this with rolling seasonal baselines; for hand-rolled monitoring, you can approximate it with day-of-week and hour-of-day buckets.
Where should the data quality monitoring live: in the pipeline itself or in a separate layer?
Both, at different stages. In-pipeline testing (dbt tests, Great Expectations checks, Soda contracts) catches expected quality violations at execution time and can halt the pipeline. External data observability catches unexpected drift after the fact. Teams that pick one and skip the other typically discover the trade-off through an incident that the missing layer would have caught. The mature answer is to run both.
How does this fit with lakehouse and streaming architectures?
The four signals (freshness, volume, schema, lineage) apply to both, but the tooling differs. Batch pipelines report freshness in hours or days; streaming pipelines report it in seconds. Batch pipelines have discrete job runs to monitor; streaming pipelines have continuous throughput. The observability layer has to model both correctly; a tool designed only for batch will miss streaming-specific issues like lag growth or partition imbalance.