Enterprise
A 900-person distributor modernized without a big-bang cutover
2 days to make a pricing change, down from roughly six weeks
eCommerceCloud and DevOps6 min read
How a D2C outdoor brand took 35% off its cloud bill and stopped dreading its own peak day
Checkout had been down for nineteen minutes on the previous peak day, and the cloud bill had roughly tripled in two years — because every performance problem had been answered by making the instance bigger.
At a glance
- Client
- a direct-to-consumer outdoor equipment brand, roughly 40 staff, in the US and Canada
- Industry
- eCommerce
- Capabilities
- Cloud Solutions, DevOps, Database Management, Software Management
- Engagement
- Cloud and DevOps — time and materials
- Duration
- 5 months, landing eleven weeks before peak
- Team
- 3 engineers, 1 architect, fractional SRE
- Built with
- Next.js, React, Node.js, PostgreSQL, AWS
What changed
3693360455%
off the monthly cloud bill, after the architecture changed rather than before
14711.04899s
mobile largest contentful paint on the top twenty category pages, from 4.8s
36933.26011×
the previous year's peak order rate carried with no checkout errors above baseline
Figures are as reported by the client over the period named in the body below, and were not independently audited by us.
Working on something like this?
Get an estimateThe short version
- The bottleneck was the catalogue query, and nobody had looked at it. Every category page ran the same uncached query against the primary database.
- Right-sizing an architecture you have not fixed only locks in the wrong shape more cheaply, so the cache and read replica went in before any instance was resized.
- The storefront cut over category by category, so the first week of real traffic was a test rather than a launch.
- Load testing ran against the previous year's actual traffic curve, spike shape included, not a synthetic ramp no customer produces.
The problem
Checkout was down for nineteen minutes on the previous peak day. The cloud bill had roughly tripled in two years. Those two facts were being treated as separate problems, and they were the same problem.
Every performance complaint over those two years had been answered the same way — a larger instance. That works, briefly, and it is the most expensive way to buy time. It also hides the cause, because a bottleneck that has been outgrown twice looks like a capacity issue rather than a query issue.
- Every category page ran the same uncached query against the primary database. At normal traffic it was slow. At peak it was the whole outage.
- The storefront had grown eleven plugins deep. Each added markup to every page, and none of them could be removed without someone knowing which of the eleven the merchandising team relied on.
- Mobile largest contentful paint was 4.8 seconds on the pages carrying the most traffic, which on a phone on a mixed connection is long enough to lose the session before the page resolves.
What made it hard
What we did
1
Fixed the query before touching the infrastructure
A read replica and a cache went in front of the catalogue in the first fortnight, on the existing stack. This was deliberately unglamorous and deliberately first: resizing instances around a broken access pattern buys a cheaper version of the same architecture, and then the replatform inherits it.
2
Rebuilt the storefront headless, against the commerce backend they already had
Next.js in front, the existing commerce platform behind. The backend was working, the team knew it, and the order and inventory data lived there. Replacing it as well would have doubled the project and put the system of record in the blast radius of a front-end deadline.
3
Cut over category by category, not all at once
Traffic moved one category at a time behind a routing rule, starting with a mid-volume one rather than the largest or the smallest. The first week of real traffic on the new storefront was therefore a test with a rollback measured in seconds, and the eleven plugins were retired one at a time as each category left them behind.
4
Load tested against last year's actual curve
Including the shape of the spike — the sharp arrival at the top of the hour, the sustained plateau, the long tail. A synthetic ramp finds the throughput ceiling and misses the failure that only happens when everything arrives together, which is the one that took checkout down.
5
Right-sized the fleet last, once the shape was settled
With the query fixed and the storefront rendering statically where it could, the instance sizing that had accumulated over two years was simply wrong in the other direction. That is where most of the 35% came from — not from a discount, and not from a reservation.
6
Ran a game day three weeks before peak
Their team, their on-call rota, a rehearsed rollback, and two failures injected without warning. The point was not to prove the system worked. It was to find out whether the people who would be awake in November knew what to do, while it was still September.
Why we did not replace the commerce backend
It was the obvious candidate — it was old, it was the thing the plugins attached to, and replatforming it would have been a cleaner architecture at the end. It was also where every order, price, and stock level lived, and it was not the bottleneck.
Replacing a working system of record because a different part of the stack is slow is how a five-month project becomes a fourteen-month one. The test we apply is simple: name the symptom, then name the component causing it. If those are different components, fix the second one and leave the first alone.
What changed
3693360455%
off the monthly cloud bill
14711.04899s
mobile LCP on the top twenty category pages, from 4.8s
36933.26011×
the previous peak order rate, no checkout errors above baseline
The 35% is the number a buyer notices, but the mechanism is the order of operations. The bill fell because the architecture changed first and the sizing changed second. Doing it the other way round — the standard cost-optimization engagement — would have found perhaps a third of it, and the peak-day outage would have happened again.
The constraints we worked under
What this engagement did not include
- The commerce backend was not replaced, not upgraded, and not refactored. It is the same system it was, and it will need attention on its own schedule.
- Search stayed on the incumbent vendor. It was adequate and out of scope, and swapping it during a replatform would have confused two sets of results.
- Checkout itself was not rebuilt — it was moved behind the new storefront unchanged, deliberately, so the highest-risk page was not also the newest code on peak day.
- We cannot attribute revenue to the LCP improvement. Conversion moved over the same period and so did the marketing mix, and separating them was not possible with the data available.
- One thing we would do differently: we sequenced the game day three weeks out and found two runbook gaps that took a fortnight to close properly. Six weeks out would have left room for a second rehearsal.
What they own now
- The storefront repository, in their GitHub organisation, and the pipeline that deploys it.
- The AWS account and the infrastructure as code that describes the fleet at its new size.
- The load-test suite built from their own traffic curve, re-runnable before each peak with one command.
- The game-day scenarios, the runbook, and the rollback procedure their team rehearsed and executed.
- A documented record of which of the eleven plugins did what, produced during the retirement, which did not exist before.
When this would not have worked
Headless earns its keep in specific conditions and is an expensive answer outside them. We would rather name those before quoting.
- If your storefront is a standard catalogue on a themed platform and it performs adequately, headless buys you complexity and a front-end team you did not previously need. Headless commerce, and when it earns its keep is the honest version of that argument.
- If merchandising needs to change page layout without a deploy, plan for that explicitly. It is solvable, and it is not free, and discovering it after cutover is painful.
- If peak is inside three months, do the query and caching work and stop. A replatform landing near peak is a worse risk than the problem it fixes.
- If nobody on your side will own a front-end deploy pipeline afterwards, the ownership model in this project does not transfer and a managed platform is the better answer.
If this sounds like your situation
The build capability is cloud solutions for eCommerce and the release side is DevOps for eCommerce. The engagement shape is cloud and DevOps, the stack is Next.js, and if the bill is the reason you are reading this, what actually drives cloud migration cost covers the pattern this project ran into.
A bigger instance is not a fix. It is a receipt for the fix you did not make.
“We had been told for two years that the fix was a bigger database. The fix was one query and a cache in front of it.”
Recognise any of this?
Tell us what is not working. You will get a scoped estimate and an architecture you own, not a capability deck.
Common questions
Not by itself. On this project the 35% came from fixing the catalogue access pattern and then right-sizing a fleet that had grown around the broken version — the replatform made static rendering possible, which helped, but the order of operations did the work. A headless build on top of an unfixed query would have cost roughly the same to run as the platform it replaced.
Move traffic in slices behind a routing rule rather than switching the domain. Here it was one product category at a time, starting with a mid-volume category so the first real traffic was meaningful but survivable, with a rollback measured in seconds. The alternative — a single cutover weekend — concentrates every unknown into the one night when everyone is tired.
Far enough that your team operates the new stack through at least one rehearsal and one ordinary busy week before the real one. On this project that meant landing eleven weeks out with a game day three weeks out, and in hindsight the game day should have been six. Inside three months to peak we would advise fixing the specific bottleneck and deferring the replatform.
Usually, and we did here. The backend held every order, price, and stock level, and it was not the cause of the outage. Replacing a working system of record because a different component is slow is the most common way a five-month project turns into a fourteen-month one. Where the backend genuinely is the constraint, that is a different engagement and we would scope it as one.
Replaying the previous year's own request pattern — the sharp arrival at the top of the hour, the plateau, the long tail — rather than a synthetic ramp to a target throughput. A ramp finds your ceiling. A real curve finds the failures that only happen when everything arrives at once, which on this site was the failure that took checkout down for nineteen minutes.