New: CentriCall AI voice agents that answer, qualify, and book around the clock
Centricone Technologies
EnterpriseSaaS Platform Build6 min read

How an in-app AI assistant finally left a six-month pilot, built by two engineers inside the client's own squad

Everyone liked the demo. Nobody would put it in front of a customer, because there was no evaluation set — so “better” was an opinion held by whoever had spoken last.

At a glance

Client
a B2B enterprise software vendor, roughly 600 staff, serving around 900 customer organisations, in the US Midwest
Industry
Enterprise
Capabilities
Artificial Intelligence, IT Staffing, Cloud Solutions, Cybersecurity Services
Engagement
SaaS Platform Build — embedded team
Duration
14 weeks embedded, then a continuing engineer
Team
2 engineers embedded in the client's product squad
Built with
Python, Next.js, PostgreSQL, Microsoft Azure

What changed

8148804899%

correct-and-cited on a 400-ticket evaluation set, from 61% at the start

1471159344

weeks from the embed starting to general availability, after six months in pilot

36933

cross-tenant retrieval defects found in pre-release testing, all before a customer saw one

Figures are as reported by the client over the period named in the body below, and were not independently audited by us.

Working on something like this?

Get an estimate
The short version
  • The first four weeks produced no model changes at all. They produced an evaluation set from 400 real support tickets, so every later change could be argued about with a number.
  • Retrieval was scoped per tenant. In a multi-tenant product the expensive failure is not a wrong answer — it is a right answer drawn from another customer's documents.
  • A refusal path and a citation requirement went in as product requirements, not as prompt instructions.
  • Two engineers worked inside the client's squad, on the client's SDLC and change control, rather than as a parallel team producing a handover.

The problem

The assistant had been in pilot for six months. Everyone liked the demo. Nobody would put it in front of a customer, and the reasons were never written down in a form anyone could act on.

It answered the questions it had been demoed with and invented answers to the long tail. There was no evaluation set, so every proposed improvement was assessed by someone trying twenty questions they had thought of themselves, which measures the questions rather than the assistant.

  • “Better” was an opinion held by whoever had spoken last. Two prompt changes had been reverted and then reinstated, both times on the strength of a demo.
  • Both machine learning engineers had been pulled onto a data migration, which is the ordinary fate of a pilot with no committed date.
  • Legal would not sign off, because there was no record of what the model had been shown at the moment it answered — so a customer complaint could not be investigated.

What made it hard

What we did

1

Spent four weeks building an evaluation set and changed nothing else

Four hundred real support tickets, each with the answer a competent support engineer would have given and the document it should have cited. Scored for correct-and-cited rather than for plausibility. This was the entire deliverable of the first month, and it was the change that made every later change decidable.

2

Scoped retrieval per tenant, at the index rather than in the filter

Tenant isolation enforced where the documents are stored, not as a predicate on the query that someone can forget to apply. A filter is a thing you remember; an index boundary is a thing you cannot bypass. Three cross-tenant defects were found in pre-release testing, and all three would have passed a filter-based design's own tests.

3

Made refusal and citation product requirements

Not prompt instructions. If the retrieved context does not support an answer, the assistant says so and offers the ticket path — and that behaviour has its own tests and its own line in the specification, so it survives a model change.

4

Logged model version, prompt, retrieved context, and output together

Under the retention policy the vendor already had. This is what legal had been asking for and what nobody had framed as an engineering requirement: a complaint about an answer given in March has to be reconstructable in June.

5

Embedded in the client's squad rather than beside it

Two of our engineers, on the client's SDLC, branching model, code review, and change control. Not a parallel team producing a handover — the handover in this model is continuous, and there is no week where knowledge transfers in a meeting.

6

Shipped to general availability behind a per-tenant flag

Rolled out by customer cohort rather than by date, starting with the tenants whose document sets were largest and messiest, on the principle that the hardest cohort tells you the most and the easiest cohort tells you nothing.

Why the model changes came fourth

Because until there was an evaluation set, a model change was indistinguishable from a mood. The team had already spent six months making changes that could not be assessed, and the correct read of that is not that the changes were bad — several were good — but that nobody could tell, so none of them could be defended.

Once the 400 tickets existed, the sequence became ordinary engineering: measure, change one thing, measure again. Correct-and-cited went from 61% to 89% over ten weeks, and roughly two-thirds of that came from retrieval and chunking rather than from the model or the prompt. That distribution is typical and it is not what the pilot had been spending its time on.

What changed

8148804899%

correct-and-cited on the 400-ticket evaluation set, from 61%

1471159344weeks

from embed to general availability, after six months in pilot

2582237122%

fewer tier-one tickets in the first quarter after GA

The 61% to 89% is measured on one evaluation set built from one product's support history. It says the assistant improved against the questions its own customers actually ask. It does not transfer to your product, and a vendor quoting a percentage without naming the set is quoting nothing.

The three cross-tenant defects are the number we would rather a buyer asked about. They were found because the evaluation set included deliberately adversarial retrieval cases, not because someone spotted them while using the product. Without that, the first person to find them would have been a customer.

The constraints we worked under

What this engagement did not include
  • No model was trained or fine-tuned. Every gain came from retrieval, chunking, evaluation, and product design around the model.
  • The assistant is read-only. It answers and cites; it does not take actions in the product, which was scoped out early and remains out.
  • Eleven percent of the evaluation set is still wrong or uncited at GA, and the refusal path covers most but not all of it. We would not describe that as solved.
  • We cannot attribute the 22% ticket reduction to the assistant alone. Two unrelated releases shipped in the same quarter and the client's own reporting does not separate them.
  • One thing we would do differently: the evaluation set was built from resolved tickets, which are biased toward questions that had answers. Adding a sample of abandoned conversations would have surfaced the refusal cases earlier than it did.

What they own now

  • The 400-ticket evaluation set and the harness that runs it, which is the durable asset — the model will be replaced and the set will not.
  • The retrieval service and its per-tenant index boundaries, in their own Azure tenant, deployed by their own pipeline.
  • The answer log schema and its retention configuration, mapped to the policy they already had.
  • The adversarial retrieval test cases, including the three that caught real defects.
  • The code, in their repository, reviewed by their engineers throughout — because our two engineers were in the squad rather than beside it.

When this would not have worked

An embedded team on an AI feature is a good fit under specific conditions, and a poor one outside them.

  • If there is no corpus of real questions — a new product with no support history — four weeks cannot produce an evaluation set, and the honest first step is instrumenting the product to collect one.
  • If the answer needs to be authoritative rather than well-cited, retrieval is the wrong shape. Regulatory and clinical answers want a decision system with a human in it, not an assistant.
  • If your squad cannot absorb two external engineers into its own review and release process, the embedded model produces friction rather than transfer, and a scoped project team is better. Staff augmentation versus a managed team covers the trade.
  • If the pilot is stalled on a business decision rather than an engineering one, an evaluation set will make the decision clearer and will not make it for you.

If this sounds like your situation

The capability is AI development services, the model is engineers embedded in your team, and the platform context is a SaaS platform build. If you are in the same place this vendor was, getting an AI pilot into production and multi-tenant architecture choices are the two pieces that matter most.

A demo is easy and a deployment is not. Build the evaluation set first, and the arguments stop being about how it felt on a Tuesday.
“For six months we argued about whether it felt better. Four weeks in we were arguing about a number instead, and those arguments finished.”
VP Product, a B2B enterprise software vendor

Recognise any of this?

Tell us what is not working. You will get a scoped estimate and an architecture you own, not a capability deck.

Get my project estimate

Common questions

Start with an evaluation set built from real questions your users have actually asked, scored for correct-and-cited rather than for plausibility. On this project that took four weeks and produced no model changes, which felt slow and was the reason the following ten weeks worked. Without it, every improvement is assessed by demo, and a pilot assessed by demo can run indefinitely — this one had run six months.
A confident, correct, well-cited answer drawn from another customer's documents. It is worse than a wrong answer because nobody notices until the wrong customer recognises their own data. Enforce tenant isolation at the index boundary rather than as a query filter, and include deliberately adversarial retrieval cases in your evaluation set — that is how the three defects here were found before release rather than after.
Not on this project, and not by default. The gain from 61% to 89% came roughly two-thirds from retrieval and chunking and the rest from product design around the model. Fine-tuning is a reasonable later step once retrieval is measured and good; doing it first spends money improving the part that was usually not the bottleneck.
Because the client had a product squad, an SDLC, and a release process that were all working, and the feature had to live inside them permanently. An embedded pair on their branching model and code review means the handover is continuous rather than an event. Where a client has no squad to embed into, a scoped project team is the better shape and we would say so.
Make the answer reconstructable. Model version, prompt, retrieved context, and output stored together under the retention policy the business already has, so a complaint about an answer given in March can be investigated in June. On this project that was the specific blocker, and it had never been written down as an engineering requirement — it had been raised as a concern, which is not the same thing.

Related work