The certificate expired on a Saturday

What actually happens

This is the only incident type where the exact date and time of the outage was known, in advance, to everyone, and it happened anyway.

A TLS certificate on a production endpoint expires. The expiry date was embedded in the certificate itself, readable by anything that connected to it, for the entire preceding year.

The renewal was tracked. It was tracked in a spreadsheet, or a calendar reminder, or a ticket queue, and the owner of that tracking mechanism changed roles at some point without the tracking changing hands.

The failure is total and immediate. Clients refuse the connection. There is no degradation phase, no partial service, and no warning at the moment it happens beyond the connection errors themselves.

It happens on a Saturday because certificates expire on the anniversary of their issue and nobody schedules issuance around weekends. The people who can renew it are not at their desks, and the process for getting an emergency certificate issued outside business hours is exactly as smooth as you would expect.

The follow-up action is invariably to build a better spreadsheet, which works until the next person changes roles.

The outage was scheduled a year in advance, in a machine-readable field, on the thing that failed. It was not a monitoring problem. It was an ownership problem.

The same renewal, two ways

There is no diagnosis time to compare here. The comparison is between an emergency and a routine task.

There is no MTTR comparison to make here, because the entire value is that the incident does not occur. The honest framing is that this converts a two-hour weekend outage into a routine change ticket.

The reason spreadsheets fail is not diligence. It is that the inventory and the ownership both drift, and a static list captures neither. Certificates get issued outside the process, endpoints get added, and people change roles.

Discovery is the mechanism that matters more than the reminder. Scanning live endpoints for what they actually present finds the certificates nobody put on the list, and those are reliably the ones that cause the outage.

Resolving ownership from current service ownership records rather than from the original requester is the second half. A reminder that arrives correctly and routes to someone who left is functionally identical to no reminder at all.

Who decides to press go

Deploying a certificate to a production endpoint interrupts connections briefly, so it is scheduled rather than executed on discovery.

Discovery, inventory maintenance, expiry tracking, ownership resolution and certificate issuance staging all run under policy. None of it touches a live endpoint.

The Action Ticket is raised and held with the endpoint list, the proposed change window, the certificate chain validation and the rollback path attached. The service owner approves and schedules.

The point is not to renew certificates automatically without anyone knowing. It is to make sure the ticket reaches a person who currently exists, thirty days before it matters.

Certificates discovered by scanning what live endpoints actually present. Expiry dates read from the certificates themselves rather than from a maintained list.

Expiry correlated against current service ownership records, endpoint inventory, the certificate authority process and the change calendar.

Action Ticket raised 30 days ahead with the endpoint list, issuance path and rollback. Held for the current service owner to approve and schedule.

Newly discovered endpoints added to the inventory automatically. Ownership re-resolved on every cycle so role changes do not orphan a renewal.

This is a platform capability, not a published customer deployment for this exact scenario. The mechanism, which is correlated investigation followed by governed MOP execution with pre-check, post-check, rollback and approval gating, is running in production today; see governed day-2 operations across 2,000+ nodes, infrastructure observability across a large server estate and closed-loop network automation. The timings shown are modelled, not measured.