No central visibility
A single request crossing four services leaves evidence in three systems. Reconstructing it means three tabs, three time pickers and correlation done by eye at two in the morning.
Three Tools, One Signal Store
Reference Architecture
How we build OpenTelemetry-native operations for Kubernetes, on-premises and in cloud. The initial state we keep meeting, the rules we write instead of the ones we find, the procedures the alerts route to, and how we would measure whether any of it worked.
The initial state we keep meeting
We rarely walk into an estate where somebody designed the observability stack. We walk into one that accumulated. Each signal type arrived with its own tool, its own agent, its own storage and its own query language, usually a year apart and usually under deadline. Nobody made a bad decision. The stack is the sum of six reasonable ones.
Six problems then show up together, and they are not independent of each other:
No scalable procedure execution
Methods of procedure live in a wiki. Running one across two thousand nodes means either a war room or a shell loop that nobody wants their name on.
Three tools, three stores
Jaeger for traces, Kibana over Elasticsearch for logs, Grafana over Prometheus for metrics. Three retention policies, three capacity plans, three upgrade cycles, three on-call skills.
Alert fatigue with no correlation
Each tool alerts on its own. One failing node produces alerts in all three. There is no deduplication layer and nothing that says these forty alerts are one event.
Thresholds nobody can change
Thresholds sit inside dashboard JSON and rule files across several repositories. Changing one costs a pull request, a review and a deploy, so nobody changes them.
Everything needs an engineer
A new dashboard, alert rule or log analytics view needs someone fluent in PromQL, Lucene and the trace data model. That person becomes the bottleneck for the whole team.
Why we build it OpenTelemetry-native
The obvious response to three consoles is to buy a fourth that queries all three. We do not, because it leaves the actual defect in place.
The defect is identity. A trace span, a log line and a metric sample describing the same pod at the same moment carry different labels, produced by different agents, under different naming conventions. One calls it pod, one calls it kubernetes.pod.name, one calls it pod_name. A federating query layer papers over that with mapping rules, and the mapping drifts the moment somebody deploys a new service or renames a label. You have not removed the problem, you have hired it.
So we fix it upstream. One collection layer stamps the same resource attributes on all three signals at the point of collection, using the OpenTelemetry resource semantic conventions. After that the join is free, permanently. That is the single reason we standardise on OpenTelemetry, and it is worth more than any individual feature that comes with it.
It also decides the order of everything else. Correlation, deduplication, cited root cause and autonomous execution all depend on being able to answer "what else was happening on this resource" without a lookup table. Get identity right at collection and every layer above it gets simpler. Get it wrong and every layer above it inherits a mapping problem you will be maintaining for years.
The solution
Five layers. Each has one job, each depends only on the layer beneath it, and each is worth switching on before the next one exists. We deliberately do not ask anyone to adopt all five to get value from the first two.
Phase 1: foundational implementation
The agent goes on as a DaemonSet per node, with a collector deployment handling aggregation, tail sampling and export. It emits traces, metrics, logs and Kubernetes events over OTLP. Three decisions in this phase matter more than the rest.
We stamp resource attributes once, at the edge. The agent enriches every record with Kubernetes resource attributes read from the local API server before the record leaves the node. Cluster, namespace, workload, pod, node, container and image land identically on a span, a log line and a metric sample. There is no downstream mapping layer to maintain because there is nothing left to map.
We treat the protocol as the contract, not the agent. Anything already instrumented with an OpenTelemetry SDK sends to us unchanged. Prometheus scrape targets and structured stdout logs are handled by receivers rather than rework. Swapping our agent for a vanilla upstream collector is a configuration change, not a migration. That is deliberate on our side. An architecture you cannot leave is one you should not enter, and we would rather say that than have you discover it in year three.
We do not switch anything off. In this phase the agent exports to the existing Jaeger, Elasticsearch and Prometheus backends and to the new signal store at the same time. Every dashboard keeps working. Every alert keeps firing where it fires today. If somebody proposes turning off Kibana in week one, including us, that is the moment to push back.
What exists at the end of phase 1: one store holding all three signals under one identity, running beside everything you already have. Nothing decommissioned, nothing at risk, and a query surface where a trace, its logs and its metrics open together. That last thing is usually the moment a team decides whether the rest is worth doing, which is why we put it first.
Capacity planning also collapses from three exercises to one. Most estates we see over-provision the log tier and under-provision traces, because the two were sized independently by different people at different times, and the correction only arrives after the incident where the traces had already aged out. More detail on the store itself is on TelemetryOps.
Phase 2: rewriting the rules
Alert fatigue gets described as a volume problem. It is a structure problem. Forty alerts for one failing node is not forty pieces of information, it is one piece delivered forty times, and muting thirty-nine of them is not a fix.
Once every signal carries the same identity, we rewrite the rules. Here is the shape of the change on conditions everyone recognises. The middle column is what we typically find in place. The right column is what we write instead.
| Condition | The rule we usually find | The rule we write instead |
|---|---|---|
| CrashLoopBackOff | Container restart count above 3 in 10 minutes on any pod, page immediately. | Restarts above 3 in 10 minutes and healthy replicas below the workload minimum. If the image digest changed in the last hour it groups under the deployment, not as a new incident. |
| Node disk pressure | Disk usage above 80 percent on any node. | Disk usage above 80 percent and projected to reach 95 percent within four hours at the current growth rate and image garbage collection is not already reclaiming. |
| Memory saturation | Container memory above 90 percent of its limit. | Memory above 90 percent of limit and at least one OOMKill in the window and the limit has not been changed in the last 24 hours. |
| HTTP 5xx rate | 5xx rate above 1 percent for 5 minutes on any service. | 5xx rate above 1 percent for 5 minutes and above that service's own rolling seven-day baseline. Errors sharing a trace root group into one incident rather than one per service. |
| Ingress latency | p99 latency above 2 seconds. | p99 above the service's rolling baseline by a configured factor and request volume inside its normal band, so a traffic spike does not read as a latency regression. |
Illustrative rule shapes, not shipped defaults. Every threshold and window here is something you set for your own estate.
The same rewrite applies below Kubernetes, on the network and hardware side where most of our recorded production work sits:
| Condition | The rule we usually find | The rule we write instead |
|---|---|---|
| Interface flap | Link down trap, page. | Link state transitions above a threshold in a window and the peer interface reports the same and no change record is open on the device. |
| Hardware sensor | Temperature or voltage above the vendor threshold, raise a ticket. | Above threshold and rising across the last several samples and the redundant unit is not already carrying the load. |
| Routing session down | Session down, page. | Session down and the affected prefixes have not reconverged via an alternate path inside the hold window. |
Three properties make those rules writable at all, and each is worth naming.
An operator writes them, not a specialist. The rule and formula builder expresses a condition over any signal in the store without PromQL, Lucene or a trace query. A rule spanning signal types is one rule here and three tools plus a human in the accumulated architecture. The formula builder handles ratios, rates of change, comparison against a rolling baseline and composite health scores, and a formula output is itself a signal, so it can be charted, alerted on, and used inside another formula.
Thresholds live in one place with an owner and a history. Not in dashboard JSON, not spread across repositories. An engineer changes a threshold for a workload or an environment without a deploy, and the change is attributable. This is behavioural rather than technical: when changing a threshold costs a pull request and a release, wrong thresholds survive for years and the team learns to ignore what they produce.
Correlation happens before anyone is paged. Deduplication collapses repeats of the same condition on the same resource. Correlation groups distinct conditions sharing a resource, a dependency path or a time window into one incident with one owner. Because identity is already on every record, this is a grouping over attributes that exist, not an inference over a mapping table. The forty alerts still exist and are still queryable. They are not forty pages.
Phase 3: the procedure the alert routes to
A correlated incident with a cited root cause is still a description of a problem. Phase 3 is where it acquires an action. A method of procedure that exists only as a wiki page is not an operational capability, it is a description of one.
Every step the engine runs is bracketed. A pre-check asserts that the conditions the step assumes are still true at the moment of execution, because the state that justified the plan may have moved between reasoning and running. A post-check asserts that the intended effect actually happened. Rollback is armed before the step executes, not written afterwards. A step whose post-check fails does not quietly continue to the next one.
Runs are parallel and pod-based. The engine sizes a run by starting execution pods appropriate to the work, and each pod handles many targets. A run across two thousand nodes starts tens of pods, not two thousand. Pod count is a function of batch size, concurrency limits and the blast radius policy for the action, never a function of the device count. Sequential execution across a large fleet is why patch windows turn into war rooms, and this is the part that removes it.
These are procedures from our shipped library, matched to the correlated condition that routes to them. We list the slugs because a claim you can go and look at is worth more than one you cannot.
| Correlated condition | Procedure | What it does | Radius |
|---|---|---|---|
| Workload unhealthy, replicas below minimum | sentinel-k8s-self-heal | Restart the workload, let the scheduler reschedule unhealthy pods, clear failed pods. | One workload in one namespace. |
| Node hardware or disk failing | sentinel-node-cordon-drain | Cordon and drain the node so workloads reschedule onto healthy ones. | Moves live traffic. Human gate. |
| Live config differs from Git | sentinel-config-drift-reconcile | Sync the app back to its Git desired state through ArgoCD. | Whatever the diff contains. Human gate. |
| Sustained saturation inside cost guardrails | sentinel-cluster-scale | Scale the workload to the target replica count under FinOps guardrails. | Cost impact. Guardrailed. |
| Certificate approaching expiry | sentinel-cert-rotate | Pull the new certificate and key from Vault, update the Kubernetes TLS secret, roll workloads. | Rolls workloads. Human gate. |
| Canary analysis returned a verdict | sentinel-canary-promote | Promote or abort the Argo Rollouts canary on the analysis result. | Release gate. |
| Post-check regressed after a change | sentinel-argocd-rollback | Roll the app back to the last-good revision. | This is the rollback path. |
How we would measure whether it worked
This is the section where a case study would put results. We do not have a Kubernetes telemetry consolidation to report, so instead here is exactly what we would instrument, which is the part most write-ups leave out anyway.
Take the baseline before you change anything. Four numbers, captured for at least four weeks of normal operation before phase 1 goes on: alerts delivered per on-call engineer per week; distinct alerts per incident that turned out to be one event; MTTR split into detect, assemble, decide, act and verify rather than as a single figure; and the count of incidents where the first responder could not act and had to escalate. Without that baseline, every number afterwards is an assertion.
Run correlation in shadow before it pages anyone. In phase 2 the correlation engine runs in parallel with the existing alert path for several weeks and pages nobody. You then compare what it would have delivered against what actually got delivered, and against which of those turned out to matter. That comparison is the evidence for cutting over, and it is produced before you take any risk.
Measure to verified closure, not to procedure completion. This one we learned on our own platform. A procedure that finished is not an incident that ended, and an MTTR measured to the last step of the run flatters itself. The honest figure stops when the original condition has been re-tested and passed. Expect that number to be worse than the one you were quoting, which is the point of measuring it.
Be careful what you optimise. Total alert count is the easiest metric to move and the easiest to move dishonestly, because muting achieves it. Watch the ratio of alerts to real incidents, and watch escalation rate, which goes up if correlation is grouping things that are not actually one event. A noise reduction figure with no false-negative figure beside it is half a result.
What this does not solve
It does not fix unusable instrumentation. If services emit no traces and log unstructured text with no correlation identifier, the collection layer will faithfully collect unusable data. Application instrumentation is real work and this does not remove it, though standardising on OpenTelemetry means it is done once against an open specification instead of once per vendor.
Correlation over a wrong topology is confidently wrong. Where dependency topology is derived from traces it is generally accurate. Where it comes from a stale configuration database, the grouping inherits the staleness, and a wrong grouping is worse than no grouping because it looks authoritative.
Autonomy is bounded by the quality of the checks. An execution engine is exactly as safe as its pre-checks and post-checks. A procedure whose post-check confirms only that a command exited zero has verified nothing about the system. Writing genuinely discriminating checks is the hard part of automation, and it is not itself automatable.
Consolidation is a migration, not a switch. Dashboards, runbooks, muscle memory and on-call habit are all built around the tools you have. The parallel-run phases exist because that does not go away by being ignored.
Looking ahead
The direction this points is not a better console. It is operations that spends its attention on the incidents that need judgement, because the ones that do not have already been correlated, explained and either handled or handed over with the plan attached. The signal store is what makes that possible, the rules are what make it trustworthy, and the checks are what make it safe. In that order, and not in any other.
Frequently asked questions
A reference design. No number on this page is a measured result. Our measured deployments are published as case studies with a named scope and a stated basis of measurement for every figure, and none of the current set is a Kubernetes telemetry consolidation. We would rather label this correctly than borrow numbers from a different deployment on different infrastructure.
Not to start, and possibly not at all. The agent exports to your existing backends and to the signal store simultaneously, so phases 1 and 2 run alongside current tooling. Some teams keep Grafana permanently as the visualisation layer with the signal store underneath it. The architectural claim is about one collection path and one store, not about which console people look at.
No. The agent speaks OTLP, so anything already instrumented with an OpenTelemetry SDK sends to it unchanged. Prometheus scrape targets and structured stdout logs are handled by receivers rather than rework. That is the point of standardising on the protocol rather than on an agent.
Not today. Every procedure in the current library carries an approval requirement. Autonomy is granted per action, on the evidence from running that action with approval first, and it is gated on the blast radius of the action rather than on the confidence of the diagnosis. The governance model is written up in what an action ticket is.
Pod count is derived from the run, not the target count. The planner sizes the run against batch size, concurrency limits and the blast radius policy for the action, then starts that many execution pods, and each pod processes many targets in sequence. A run across thousands of nodes starts tens of pods. There is a worked example on the ProcBot day-two operations case study.
Yes. The collector, the store, the alert engine and the execution engine all run inside the customer perimeter, and the reasoning layer can run on customer infrastructure with no egress. The tradeoffs of that deployment model are on air-gapped AIOps.
One cluster, agent deployed, exporting to both the existing backends and the signal store, with existing alert rules untouched. That gives you a cross-signal query surface for one cluster with nothing switched off and nothing at risk, and it is enough to test whether the identity claim holds in your estate. If it does not, you have lost a week and learned something useful.
Related reading: TelemetryOps for the signal store itself, agentic AIOps for the agent architecture above it, and the AIOps platform comparison for where this sits in the category.