The evals problem: how financial firms actually measure AI in production.

Ask a financial firm how they know their AI system is doing what it should, and you will hear one of three answers. The first: we look at it. The second: we have a QA process. The third, from the mature ones: we have an evals dashboard. Only the third answer describes a system that will survive a regulator conversation or a model upgrade. The first two describe systems that will not.

Evals — short for evaluations — are the most under-explained piece of AI engineering in financial services. Every serious deployment we ship has one. Every failed deployment we have been called in to fix, at some point in the diagnostic, turns out to be missing one. This note is what an evals layer actually looks like, why it matters, and how much it costs to build.

What an eval is, precisely

An evaluation is a machine-graded, versioned test of an AI system output against a rubric. The rubric is domain-specific. The test set is drawn from real production traffic. The grades are stored, timestamped, tagged with model version and retrieval version, and made available in a dashboard the engineering team looks at every day and the CCO looks at every quarter.

That is it. It is deceptively simple in shape and profoundly difficult in practice, because the rubric is where the domain expertise lives and building a rubric that reliably distinguishes good outputs from bad ones is real work. Every mature evals layer in every firm we have shipped one for went through three rubric iterations before it stabilised.

Why firms skip it

There are three reasons firms skip the evals layer, none of them defensible. First, the team believes the model provider’s benchmark scores mean the model is good. They do not. Benchmark scores describe a model’s behaviour on someone else’s data with someone else’s rubric. Your production is neither.

Second, the team believes QA is the same thing. QA measures whether the button works and the response returns. Evals measure whether the response is correct. These are different questions and require different infrastructure.

Third, the team believes evals are for large firms. This is backwards. Small firms need evals more, because one bad output at low volume is a much larger fraction of your customer experience than one bad output at high volume. Not building evals is a claim that you will notice all bad outputs manually. That claim is almost never true.

What we score, in every deployment

Rubrics vary by domain, but the categories are consistent across every project.

• Factuality against retrieved sources. Did the output stay inside its citations?
• Policy compliance. Did any output breach a policy-layer rule, whether or not the guardrails caught it?
• Task success. Did the output do the specific thing the user asked for?
• Tone and register. Is the response appropriate to the audience and channel?
• Vulnerability signals. Did the customer signal hardship, coercion or confusion, and did the response handle it?
• Suitability drift. Did the response cross the line from information into personal recommendation?

How the rubric is built

The rubric is built by a domain expert, not an engineer. It expresses each scoring category in unambiguous language, with examples of pass and fail for each. It is then encoded as a scoring prompt for a strong model that acts as the judge.

The judge is calibrated by having a domain expert score a sample of production outputs manually, comparing to the judge’s scores, and iterating until the two agree on the vast majority of cases. Once calibrated, the judge scores at scale and the domain expert only reviews the disagreements. This is the workflow that makes evals affordable to run continuously.

What evals unlock

The most obvious payoff is that you can upgrade models. Without evals, switching from GPT-4 to Claude in production is an act of faith. With evals, you re-run both against a fixed test set and make a data-backed decision. The firms with an evals layer upgrade models every eight to twelve weeks. The firms without one usually get stuck on whichever model they launched with and quietly lose ground.

The second payoff is that you can defend the system. When a regulator asks how you know it is safe, you show them the eval dashboard, the sample of human-reviewed edge cases, and the last six months of scores. Every firm we have taken through this conversation has left with a lower risk rating than they arrived with.

The third payoff is that you can debug faster. When something breaks, the evals localise the regression to a specific model version, retrieval version, or policy change. Debug time drops from days to hours. For a firm operating at scale, that is the difference between a bad afternoon and a bad quarter.

What an evals layer costs to build

For a firm with one or two customer-facing AI systems in production, a first-release evals layer is a four-to-six-week build costing £60k–£120k, plus ongoing operating cost that scales roughly with token spend on the judge model — usually 5–15% of the primary system’s inference cost. That is a small fraction of what a single production incident tends to cost.

The cheapest time to build the evals layer is at the same time as the primary system. Retrofitting adds around 40% to the cost because the test set has to be reconstructed and the rubric is negotiated in retrospect. If you are early in a build, put evals in scope now.