Evidence
Patching Two Thousand Nodes Without a War Room
Written by
Praveen Yadav
Published
August 2026
Reading time
9 min read
A Tier-1 enterprise in North America had no mass patching capability at all. The war room existed because nobody could answer one question at scale: did it actually apply?
Ask an infrastructure team why patching needs a war room and the answer is never that the commands are difficult. It is that when a rollout goes out across a few thousand machines, nobody can say with confidence what happened on each one, so a room full of people spends a night finding out.
This estate did not have a partial capability that needed improving. It had none. There was no solution for rolling out mass patching across the server estate at all. Multiple engineers did it by hand, node after node, cycle after cycle, or through scattered shell scripts that nobody governed: no approvals, no audit trail, no rollback story. Whether a patch actually applied correctly was answered node by node, if it was answered.
The bottleneck was never execution. It was verification at scale. Running a command on two thousand machines is easy. Knowing afterwards what state two thousand machines are in is the part that costs a night.
What made it unfixable by effort
Checking node health, capturing logs and reading process status across thousands of servers is not something you solve by trying harder. Neither is the governance problem underneath it. Ad-hoc shell scripting gets the work done and leaves nothing behind: no record of who approved it, no evidence of what ran, and no way to reverse it if the answer turns out to be wrong.
And the trajectory made it worse rather than better. The estate was heading toward a hundred thousand nodes on a process that was already straining at two thousand. Patching two thousand machines by hand is a staffing problem. Patching a hundred thousand is not a staffing problem, it is an impossibility, and no amount of hiring converts one into the other.
Making the MOP the unit of work
ProcBot became the day-2 execution layer for the estate, reaching nodes over SSH under RBAC with every session accountable. Software upgrades, process upgrades and patch applies run as MOPs executed across the estate, so a rollout is an operation rather than a project. Health sweeps run continuously, two thousand nodes checked the way one used to be. Node logs are captured at scale and analysed by Sentinel, so patterns surface instead of sitting unread on disk.
The structural change is the one in the middle of every run. Each MOP validates the node before touching it and verifies the result after: applied properly or not, with errors flagged and what happened recorded. Success rates, failure rates and per-node outcomes compile automatically, which is the difference between a patch cycle being a report and a patch cycle being an investigation.
What was measured
| Figure | What it is |
|---|---|
| 2,000+ nodes | Live today under governed ProcBot day-2 automation. Countable, not estimated. |
| 100% of MOP runs carry precheck, postcheck and audit | Every run, not a sampled subset. This is the governance claim and it is binary. |
| ~70% faster mass patch rollout | Against the prior manual, engineer-by-engineer approach. |
| ~60% less engineer effort on routine day-2 operations | Time returned from health walks, log collection and process checks. |
| ~100,000 nodes | Forecast at target scale, not a result. Labelled as a forecast everywhere it appears, and excluded from every headline. |
These belong to a Tier-1 enterprise in North America. Our two telecom deployments have their own numbers on their own pages, and the three sets are never stacked in one sentence. That rule, and the occasion we broke it, is in how we decide what to publish as a number.
Why the governance was the feature, not the tax
It would be easy to read RBAC, approvals and audit as overhead bolted onto the useful part. In this deployment they are the reason the useful part was allowed to exist. An estate that had been burned by ungoverned shell scripts does not respond to a proposal for faster ungoverned shell scripts. It responds to a proposal where every touch is accountable.
The precheck and postcheck pair does the same job at a smaller scale. A precheck records the state the node was in. A postcheck asserts the state it should now be in. Together they turn "the command exited zero" into "the change landed on this node and here is the evidence," which is the distinction that separates execution from remediation and the one we set out in what an Action Ticket is.
The full before-and-after across six dimensions is in the case study. The other two deployments in this series are a 27,000 device NOC in India and a North America telecom closed loop.
Frequently asked questions
Which deployment produced these numbers?
One: a Tier-1 enterprise in North America, not a telecom operator. This matters because we also publish figures from two Tier-1 telecom estates, and the three sets are different customers on different continents doing different work. They do not get added together, and the reason is set out in our evidence policy.
Is the 100,000 node figure real?
It is a forecast, it is labelled as one, and it does not appear in any headline. What is live today is more than 2,000 nodes under governed day-2 automation. The point of stating the forecast at all is that the target scale is the same MOPs rather than a proportionally larger team, which is the actual argument. Treat it as a plan, not a result.
What does a precheck actually check here?
It validates the node before anything touches it, and the run records the result. That is what makes the postcheck meaningful: without a recorded starting state, verifying the outcome tells you the command completed rather than that the change landed correctly. In this deployment 100 percent of MOP runs carry precheck, postcheck and a full audit trail.
How is this different from Ansible or a configuration management tool?
Those execute well and we integrate with them rather than replace them. What they do not supply on their own is the governance envelope: RBAC on access, approvals on execution, a precheck and postcheck bound to each run, and an audit record that compiles into a rollout report. The gap this deployment closed was not the ability to run a command on 2,000 machines. It was knowing, afterwards, what happened on each of them.