Selected engineering work · Distributed systems
Designing a live rewrite before it can fail
Four months of investigation from a vague ask to a low-level design for consistency, concurrency, and crash recovery, plus a feature-flagged rollout, so a live control plane could be replaced without corrupting state.
This case study represents work previously performed by a member of our engineering team before the formation of our consulting firm. It was delivered at Microsoft. Customer names and certain implementation details have been omitted for confidentiality.
Context
A rewrite of the brain of a production system does not start at git init. It starts at three questions nobody wants to answer while the old system is still taking traffic:
What happens to in-flight work? What happens when two sides of the system disagree? What happens when a machine dies in the middle of a write?
This work was four months of investigation, from a vague ask and no existing design, to a low-level design that answered those questions for a live control-plane re-architecture, plus a feature-flagged rollout plan the team could actually execute.
The challenge
A public rollout of a control-plane re-architecture was blocked. The tenants on the platform could not absorb a best-effort migration. The team knew the old control plane had to be replaced. Nobody had specified the distributed-systems behaviour that would make the new one safe.
The ask was vague. There was no design to extend. Hyperscale customers were the reason the rollout could not be sloppy; they are not named here.
The complexity
- State has to stay true across partitions. A control plane that disagrees with itself will eventually bill, place, or delete the wrong thing.
- Concurrency is not optional. Two requests will hit the same logical object at the same time. “Last write wins” is not a strategy at this layer.
- Crashes are normal. Recovery has to be safe across partition boundaries, not only on a happy-path restart.
- A design that cannot roll out is a paper design. The same document had to include how this turns on behind flags, and how it turns off.
Our approach
- Investigate until the design is boring. Four months of technical investigation, independently driven, instead of jumping to implementation from a slogan.
- Write the low-level design, not a vision deck. Distributed state consistency, cross-partition concurrency control, crash-safe recovery across partition boundaries.
- Align the team before writing the dangerous code. The design had to survive review by the people who would operate it.
- Rollout is part of the design. Feature-flagged strategy, defined up front, now the path implementation is following.
The solution
This page is honest about status: the design shipped; the implementation is in progress. That is still the valuable artefact. A blocked public rollout had a specified path. The team had something to implement other than hope.
Outcome
Team-wide alignment on a low-level design that unblocked the re-architecture work. A feature-flagged rollout strategy in active implementation. No claim that the entire control plane has already been replaced.
Technology
Distributed systems · cloud control plane · consistency and concurrency design · crash recovery · feature-flagged rollout · C# / .NET
Why this is here
This is prior individual work at Microsoft, not a client of this firm.
A rewrite of something important starts with the questions above, not with a quote off a slide. If the work does not need that level of design, a smaller slice is the right recommendation.
Next step
Let’s talk about what you’re building.
Whether you are planning a new product, modernizing an existing platform, or facing a complex technical challenge, we’d be happy to start with a conversation.
Start a conversation