How to Monitor CI/CD Pipelines with OpenTelemetry

Monitoring CI/CD pipelines with OpenTelemetry means tracing pipeline runs and their stages as spans over OTLP, so build and delivery health, and its bottlenecks, become observable..

Monitoring CI/CD pipelines with OpenTelemetry means tracing pipeline runs and their stages as spans over OTLP, so build and delivery health, and its bottlenecks, become observable.

What to observe in a CI/CD pipeline

A slow or flaky CI/CD pipeline taxes every engineer, but pipelines are often a blind spot. Treating a pipeline run as a distributed trace, with each stage, job and step as a span, exposes where time actually goes, which steps fail, how flaky the pipeline is, and how long work waits for a runner. The signals that matter are pipeline duration and its trend, stage and job failure rates, flaky-test rate, and queue or runner wait time.

The most useful reading is duration broken down by stage: a pipeline that has slowly crept from five minutes to fifteen almost always has one or two stages responsible, and the span breakdown names them.

How OpenTelemetry collects it

OpenTelemetry has a growing CI/CD observability effort, and many CI systems can emit trace data or be instrumented to. You instrument the pipeline so each run produces a trace with spans for its stages, jobs and steps, including durations and outcomes, exported over OTLP, and collect build and test logs with the filelog receiver for detail.

Because CI/CD tracing conventions are still maturing, the practical approach is to emit a span per stage and step with consistent attributes (pipeline, branch, result) so you can aggregate across runs; that consistency is what turns individual traces into trend data.

CI/CD span and metric conventions are still evolving in OpenTelemetry; the signals below are derived from per-run spans.

Key metrics to watch

The signals that explain most pipeline problems, derived from per-run traces:

What to put on a dashboard

A pipeline dashboard should make slow and flaky obvious. Row one: pipeline duration trend and a breakdown of duration by stage. Row two: failure rate by stage and flaky-test rate. Row three: runner queue wait time and success rate by branch. Together these tell you whether to fix a slow stage, a flaky test, or add runner capacity.

# pipeline duration trend (are runs getting slower?) avg(pipeline.run.duration) by (pipeline) # slowest stages (where the time actually goes) avg(pipeline.stage.duration) by (stage) # runner wait time (capacity bottleneck, not pipeline logic) avg(pipeline.queue.wait_time) by (pipeline)

Reading the signals: common failure modes

Slowly growing pipeline

Total duration creeps up over weeks until the pipeline is a bottleneck. The stage-duration breakdown shows which one or two stages grew, dependency install, a test suite, an image build, so you fix the actual culprit rather than optimising blindly. Caching and parallelisation usually target the identified stage.

Flaky tests

Runs fail intermittently and pass on retry, eroding trust in the pipeline until people ignore failures. Tracking the flaky-test rate across runs surfaces the specific tests. Quarantine and fix them; a pipeline whose failures are ignored is worse than no pipeline.

Runner starvation

Runs sit waiting before they even start, so total lead time is high even though the pipeline itself is fast. Queue-wait time is the signal, and it points at runner capacity rather than pipeline logic. Add runners or tune concurrency.

Example alert conditions

Starting thresholds to tune:

From monitoring to autonomous resolution

Opstral's DevSec Ops and Telemetry Ops capabilities observe delivery pipelines and can act on failures and risk through governed change gates and Action Tickets. Explore DevSec Ops and Telemetry Ops.