Changing a live platform without taking it down

Context

Most companies do not have a cloud-SKU problem. They have a we have to move, and we cannot go down on Tuesday problem.

This work sat on a large production compute platform. Tenants were already live. Strategic customers needed to move off an older clustering model onto a newer scale model without a service interruption. There was no existing “migration button.”

The shape of the engagement is the same on a 40-person SaaS, a multi-branch operations system, or a legacy app a company is afraid to restart.

The challenge

The feature had to be designed, tested, and rolled out across every production region, and it had to fail closed. A handful of important customers would not accept a maintenance window. Someone had to design the move, prove it, turn it on region by region, and still be the person who gets the call when it misbehaves.

The same engineer was also primary on-call for that compute infrastructure, and later picked up a production defect that had no owner and no documentation: a class of machines on which every update was blocked, forcing a partner team to roll back.

Separately, two storage capabilities existed on paper and were not actually usable on the paths customers needed: the boot disk, and cached reads for read-heavy workloads. That is a last-mile problem, not a brochure problem.

The complexity

1. The service cannot bounce. A migration that needs a maintenance window is not a migration these customers can use.
2. The blast radius is the whole platform. A bad rollout is not one tenant. Staged rollout, feature flags, and a test strategy that matches production are the product.
3. Nobody owns the fire. The update-blocker had no owner and no docs. Waiting for a tidy brief was not an option.
4. A capability that is not wired through does not exist. If the faster or cheaper disk tier cannot be used as a boot disk, customers do not have a faster or cheaper boot disk.

Our approach

• One owner, end to end. Design, implementation, test strategy, staged rollout across production regions, and the cross-team coordination the rollout required. About nine months on the migration feature, as the sole engineer on it.
• Prove it before you offer it to everyone. Scenario tests first; feature flags; regional stages; a way to stop.
• Treat the unowned incident as a design problem. Systematic scenario testing showed gaps in both input validation and backend placement logic. Fixes went out the same way as the migration: flags, then stages.
• Close the last mile. Compute-side design and implementation so the newer disk tier could actually be used on the VM boot path, and so cached reads could be turned on for the high-performance data disks that were supposed to support them.
• Live with the consequences. Primary on-call for the same infrastructure.

The solution

• A zero-downtime migration path, designed and rolled out across production regions, so live tenants could move clustering models without a maintenance window.
• An unowned production blocker diagnosed and fixed; a latent post-rollout failure caught in testing rather than in customers’ night.
• Boot-path and cached-read paths enabled for storage that had been stuck behind a platform gap.
• On-call coverage on the live system.

Outcome

The migration existed as a real, production path, not a wiki page. Customers who could not take downtime had a way to move. A partner team that had been forced to roll back could go forward. Storage that had been theoretically available could actually be selected.

We do not put a dollar figure on “downtime avoided.” The honest version is enough: the work was done so customer trust was not the price of the migration.

Technology

C# · .NET · large-scale cloud compute · feature flags · staged regional rollout · production on-call · platform security hardening

Why this is here

This is prior individual work at Microsoft, not a client of this firm, and we are not an Azure reseller.

It is the same class of problem we take now: a system that is already in production, a change that cannot be a Friday-night cutover, and a rollout that has to be reversible.