The financial AI vendor scorecard: four questions that expose weak products.

Finance-firm CTOs and COOs face a difficult comparison: decks can look similar, demonstrations often cover the same use cases and the phrase financial-services-grade is not a technical specification.

A useful evaluation therefore tests the system behind the interface: the evaluation harness, failure mechanism, data and control boundary, and the vendor’s willingness to reject unsuitable use cases.

This note is a fifteen-minute filter built around four questions. It is a due-diligence framework, not a ranked client study or a claim about undisclosed vendor engagements.

Question 1 — show me the evals harness

Not the demo. The actual harness. The test set, the scoring rubric, the version log of runs against it. A vendor without one is not a serious AI vendor. They are a design agency with an API key. The test set should be domain-specific — if they cannot show one, the domain claim is marketing.

Serious vendors will hand you a document, or better a live dashboard, showing every model version they have shipped, scored against a fixed rubric, with the deltas explained. The unserious ones will describe a QA process. Those are different things. QA measures whether the button works. Evals measure whether the answer is right.

Question 2 — what happens when the model gets it wrong

The right answer is a specific mechanism. Human-in-the-loop routing above a confidence threshold. Fallback behaviour to a deterministic system. Incident logging and post-mortem process. Named severity levels. A shrug and a claim about model accuracy is the wrong answer — that vendor has never operated the system at real volume.

This question filters brutally. Most vendors give an aspirational answer because they have not yet operated a customer whose failure is expensive. If you are a regulated financial firm, you cannot be their first.

Question 3 — show me the compliance envelope

Which data goes where. Which model provider sees what. How the vendor handles right-to-erasure, data residency, model-version transparency. How they evidence outputs to a regulator, including yours. If they hand you a security questionnaire and a SOC 2 report instead of an architecture diagram, you have your answer.

Serious vendors have a per-customer compliance envelope. The good ones will let you inspect it. The best ones will let your CCO reshape it before signing.

Question 4 — tell me about the last customer you turned down

This one is the killer. Serious AI vendors turn firms away — because the use case is not ready, because the data is not there, because the accuracy bar cannot be cleared, because the regulator posture is wrong. Vendors that say yes to everyone are selling revenue, not systems.

The answer to this question is a proxy for whether the vendor has an engineering culture or a sales culture. Both can be legitimate businesses. Only one of them should be embedded inside your customer-facing stack.

How to score the answers

Score the evidence, not the confidence of the presentation:

• Four evidenced answers: proceed to technical, security and commercial diligence.
• Three evidenced answers: identify whether the missing control is material to the proposed workflow before proceeding.
• Two evidenced answers: restrict consideration to low-consequence uses unless the gaps are remediated and verified.
• One evidenced answer: the product is not ready for a material regulated workflow.
• No evidenced answers: stop the evaluation rather than treating a demonstration as assurance.

When to buy and when to build

Buy when the use case is horizontal, non-customer-facing, and someone else is going to invest more in the roadmap than you can. Meeting-note summarisers, internal knowledge-base search, developer copilots — all buy. The vendor’s advantage compounds and you are unlikely to build a better version.

Build when the use case is customer-facing, sits inside your compliance envelope, or is where you intend to compete. Research substrates, KYC copilots, trader onboarding, personalised communications — all build. A shared vendor cannot be your differentiator. If it is shared, every competitor has it.

Hybrid — buy the substrate, build the controlled surface — is often a credible shape. Buy commodity model, storage and observability components; build the policy layer, retrieval curation, customer surface and evaluation rubric where the firm needs ownership.

The three vendor categories most firms get wrong

First, the compliance-labelled RAG vendors. Two-thirds of the compliance-grade vendors in the cohort do not actually enforce policy above the model — they just log. Logging is not a control. If the model can produce an out-of-policy output and the vendor’s only response is to record it, that is not compliance-grade.

Second, the horizontal AI platforms with a financial-services skin. The skin is real; the domain depth is usually not. Three months in, the firm is doing all the domain adaptation itself, at platform prices, and the vendor is not helping.

Third, the AI-driven trading tools that are actually decision-support tools. This is a definitional problem more than a technology problem — but many firms buy them thinking they are getting one thing and get another. Read the contract, not the marketing.

How to run the fifteen-minute test

You do not need an AI architect in the room to run this. You need the four questions, a notepad, and someone willing to stay in the awkward silence when the vendor pivots to a demo. The demo is not the answer to any of the four questions. Do not accept it as one.

If the vendor refuses to answer any of the four questions in writing after the meeting, that is your answer. If they answer three and struggle with the fourth, ask for a technical follow-up with the person who owns that part of the system. If they cannot produce that person, that is also your answer.

The return from this test is avoiding commitments before the operating evidence is visible. Any projected saving should be calculated from the firm’s actual shortlist, contract values and replacement costs.