Enterprise
A 900-person distributor modernized without a big-bang cutover
2 days to make a pricing change, down from roughly six weeks
TelecommunicationsCloud and DevOps6 min read
How a regional carrier ported 18,000 numbers over one weekend and held 99.9% availability
Provisioning was split across three vendor portals and reconciled by hand. Then an acquisition arrived with a bulk port-in attached, and the last one had taken a week of nights and dropped service for two hundred customers.
At a glance
- Client
- a regional competitive carrier in the US Midwest, roughly 150 staff, in the US Midwest
- Industry
- Telecommunications
- Capabilities
- Telecommunications Services, DevOps, Cloud Solutions, Database Management
- Engagement
- Cloud and DevOps — time and materials
- Duration
- 7 months, then an ongoing support retainer
- Team
- 4 engineers, 1 architect, 1 SRE
- Built with
- Python, Node.js, PostgreSQL, Kubernetes, AWS
What changed
9259904899.16199%
service availability across the port-in weekend
47044hrs
order to activate on the automated flow, down from four to six business days
36933of04899
port batches rolled back and replayed during the weekend, with no customer-visible impact
Figures are as reported by the client over the period named in the body below, and were not independently audited by us.
Working on something like this?
Get an estimateThe short version
- The activation delay was not network time. It was a person re-keying one order into three vendor portals, in sequence, during business hours.
- One flow was automated first — business voice line add, 71% of order volume — rather than all of them at a tenth of the depth.
- Provisioning jobs were made idempotent and put behind a replayable queue, so a failure is retried by the system rather than by a human at two in the morning.
- The port ran in nine batches with a rollback per batch. Three batches used it. That is the number worth asking about.
The problem
An order took four to six business days to activate, and almost none of that was network time. The switch could provision a line in minutes.
What took days was a person opening three vendor portals in sequence and typing the same order into each, during business hours, in an order that mattered because the second portal needed an identifier the first one issued. An order placed on Thursday afternoon reliably activated the following Wednesday, and nobody could explain that to a customer in a way that sounded like a reason.
- Three systems of record and no agreement between them. Billing believed one thing, the provisioning portal another, and the CRM a third. Monthly reconciliation was three days of work for one person.
- Failures were invisible until a customer called. A provisioning job that half-completed left no alert, only a line that did not work.
- A bulk port-in of roughly 18,000 DIDs was already scheduled, arriving with an acquisition. The previous one, at their old vendor, had taken a week of nights and dropped service for something over two hundred customers.
What made it hard
What we did
1
Mapped order to activate before automating any of it
End to end, with timestamps, across a quarter of real orders. The result was not what anyone predicted: the flow people complained about most was not the flow that carried the volume. Automating by complaint would have optimized the wrong path.
2
Automated one flow first — business voice line add, 71% of orders
One seam, taken all the way to done, rather than six seams taken to a demo. The remaining flows kept their manual path unchanged and untouched, so nothing regressed while the first one was proven against real traffic.
3
Made every provisioning job idempotent and replayable
Each job can run twice without doing damage, and each sits behind a queue that retries with backoff. This is the difference between a failure the system handles and a failure a human handles at two in the morning — and in telecom, the second kind is most of the operational cost.
4
Reconciled continuously instead of monthly
A scheduled job compares billing, provisioning, and CRM every night and raises the differences as tickets. The monthly three-day reconciliation became a queue that is usually short, because a discrepancy found the next morning is one conversation and a discrepancy found five weeks later is an investigation.
5
Dry-ran the port in full, then batched it with a rollback each
The whole 18,000 was rehearsed against a staging tenant, twice. The live event was split into nine batches, each with its own rollback rather than one rollback for the whole night — so a bad batch is a bad batch, not a bad weekend.
6
Wrote the runbook for their NOC, and let their NOC run it
In their language, against their alert console, rehearsed with their staff. We were on the bridge. They ran the port. That distinction is the whole point of the handover.
Why per-batch rollback mattered more than the automation
The automation is what made the weekend possible. The batching is what made it survivable. Three of the nine batches hit an error at the losing carrier and were rolled back and replayed — under a single-rollback plan, the first of those three would have meant reversing the entire event and rescheduling with the regulator. Instead each was a twenty-minute detour that no customer experienced.
This is why we would rather a buyer asked about the three rollbacks than about the 99.9%. The availability figure is the outcome. The rollbacks are the mechanism, and they are the part that transfers to your project.
What changed
9259904899.16199%
service availability across the port weekend
47044hrs
order to activate, from four to six business days
25822hrs
monthly reconciliation, from roughly three days
The activation figure applies to the automated flow, which is 71% of order volume. The remaining 29% still runs the manual path and still takes days. Quoting four hours as though it covered every order would be the kind of number that reads well and does not survive a follow-up question.
Reconciliation collapsed because the differences stopped accumulating, not because anyone got faster at reconciling. A nightly comparison turns a monthly investigation into a short daily queue.
The constraints we worked under
What this engagement did not include
- Twenty-nine percent of order volume was never automated. Those flows are low-volume and highly variable, and automating them would have cost more than the manual handling does.
- The three vendor portals are still the systems of record. We automated the path between them; we did not replace any of them, and a vendor API change is still a change we have to absorb.
- Billing rating rules were left alone entirely. They are correct, they are understood by two people, and rewriting them was not on the table.
- We cannot separate the availability figure from the acquisition's own migration work, which ran in parallel. The 99.9% is the carrier's number for the weekend, not an attribution of cause.
- One thing we would do differently: the nightly reconciliation raises tickets, and for the first month it raised too many, because we set the tolerance from the specification rather than from a week of live data. We tuned it after the noise, which is the wrong order.
What they own now
- The repository and the provisioning services, in their own accounts, deployed by a pipeline their engineers run.
- The Kubernetes platform and its infrastructure as code, sized and documented for their load rather than ours.
- The port runbook, rehearsed twice in staging and executed once live by their NOC.
- The reconciliation job and its tolerance configuration, tuneable without a deploy.
- An on-call rota and alert set their team wrote with us and now maintains alone.
When this would not have worked
Automating order to activate pays off when the volume is real and the flow is repeatable. Several situations make it the wrong first move, and we would say so before quoting.
- If no single flow carries a substantial share of volume, there is no seam worth taking all the way to done, and the work fragments into six half-finished paths.
- If your vendors expose no API — some regional portals genuinely do not — the honest answer is screen automation, which is fragile, or a vendor conversation, which is slow. We would not sell the first as a platform.
- If a port is inside eight weeks. A full dry run and a rehearsed runbook do not compress, and we would rather decline than rush a regulated cutover.
- If the reconciliation differences are mostly rating disputes rather than data drift, a nightly comparison surfaces the argument without settling it.
If this sounds like your situation
The capability is described on the telecom software development page, the platform work on cloud and DevOps, and the delivery practice behind the rollback design in what DevOps maturity actually looks like. If you are a carrier or MSP weighing this, our telecommunications work is the place to start.
In telecom the failures are in the seams. Automate the seam that carries the volume, and rehearse the one that carries the risk.
“Three batches rolled back and nobody phoned me. On the last port I was on the bridge until Sunday afternoon.”
Recognise any of this?
Tell us what is not working. You will get a scoped estimate and an architecture you own, not a capability deck.
Common questions
It depends almost entirely on whether one flow carries most of your volume. Here, business voice line add was 71% of orders, so a single seam taken to production changed the operational picture within a quarter. Where volume is spread evenly across a dozen flows, the same budget buys a dozen partial automations and the payback is much further out — which is a reason to map the flow before scoping the work, not after.
The port is split into groups of numbers, each cut over and verified independently, with a rehearsed procedure to return that group to the losing carrier if verification fails. It costs more to prepare than a single all-or-nothing plan. On this port three of nine batches used it, and without per-batch rollback the first of those would have meant reversing the whole event and rescheduling with the regulator.
Integrate, in almost every case. The portals are the systems of record and the carrier's commercial relationships run through them. What we replace is the human sitting between them retyping the same order three times. Replacing a portal is a different and much larger project, and it is rarely the one that fixes the activation delay.
Yes. The properties that matter are idempotent jobs and a replayable queue, and both are achievable on plain containers or managed services. Kubernetes was chosen here because the carrier already ran it for two other workloads and had the operational knowledge. We would not introduce it for this alone — see the honest limits on our Kubernetes page.
Their NOC, which is why the runbook was written in their language against their alert console and rehearsed by their staff rather than ours. They executed the live port. We stayed on the bridge. An ongoing support retainer covers the vendor API changes we cannot control, described on the maintenance and support page.