Why Your Pilot Is Failing Before You Choose a Vendor: The Proof of Concept Trap

TrustSphere Network

9 hours ago

7 min read

The most expensive mistake in financial crime vendor selection happens after the RFP closes. A shortlist of three to five vendors is selected, each is given a pilot opportunity, and then the institution discovers that the pilots cannot be fairly compared. One vendor's pilot used production transaction data; another used anonymised sample data. One ran for eight weeks on a subset of flows; another ran for four weeks on full-volume flows. One was led by a dedicated vendor engineer; another was given account support only. At the end, the institution has invested five person-months in pilot management, the CFO is asking why the thing is not yet deployed, and the procurement decision has devolved into a disagreement about what the vendors even attempted to prove.

The root cause is that institutions and vendors have divergent objectives in a pilot. The institution wants to know whether the vendor can operationally solve a specific problem within its own systems, at acceptable cost, with acceptable false-positive and false-negative rates. The vendor wants to demonstrate that its platform is viable in principle and that with enough integration effort (and additional consulting spend), it can work. These objectives are compatible, but only if the pilot is designed as a falsifiable experiment, not as a marketing exercise conducted in a controlled environment.

This matters operationally because vendors now know that pilots are a critical gate in procurement. Sophisticated vendors have invested heavily in pilot delivery, and the result is that a poorly designed pilot will declare a good vendor unsuitable or will declare an unsuitable vendor good, depending entirely on how the design was rigged. Institutions are learning this at different speeds, and the ones that structure pilots as true experiments, with clear success criteria, defined test scenarios, and blinded scoring, are making measurably better procurement decisions than the ones that run pilots as "figure out whether this works" exercises.

Where Procurement Pilots Go Wrong

The modal failure pattern has three stages. First, the pilot is scoped too broadly, trying to answer multiple questions at once (can the vendor integrate? can it handle our volume? can it reduce false positives? can it integrate with our secondary systems?). Answering all of these simultaneously is impossible with limited resources; the result is that each question is answered partially and incompletely. Second, the test data and test scenarios are not held consistent across vendors, so the comparisons are invalid. One vendor pilots against live transaction data; another against historical sample data. One pilots for eight weeks; another for four weeks. The comparison becomes apples-to-oranges. Third, success criteria are defined retrospectively, after the pilot results are known. Instead of "the vendor must achieve 95 per cent accuracy on entity matching", the evaluation becomes "what accuracy did the vendor achieve" and then a post-hoc decision about whether that is acceptable.

A secondary failure pattern is that pilots are run in isolation from the institution's actual operational workflows. The vendor's pilot environment does not connect to the institution's core system; it receives data through a batch feed once per day. The institution's production environment, by contrast, needs real-time decisioning. The pilot demonstrates that the vendor can work in theory; the deployment fails because the real-time integration is infeasible at the scale required.

A third pattern is pilot scope creep. The institution wants to test vendor A and vendor B. Vendor A runs smoothly on the agreed scope. Vendor B asks to expand the scope to include additional capabilities (more data sources, more list coverage, integration with secondary systems) and pitches this as "better" than the original scope. The institution agrees because the expanded scope seems more comprehensive. Now vendor B is being evaluated on a different set of capabilities than vendor A, and the comparison is invalid. Vendor B appears to offer more, but only because the pilot was designed differently.

How to Run a Pilot That Generates Actionable Results

An effective pilot follows a structured experiment design with five components.

First, a single falsifiable objective. The pilot must answer one clear question, such as: "Can this vendor achieve 95 per cent entity-matching accuracy on our corpus of existing sanctions-flagged transactions, using only the data fields that exist in our core system, and return a decision within 300 milliseconds?" Write this as a one-sentence success criterion. Everything else (integration nice-to-haves, future capability, vendor service model) is secondary and documented but not part of the pilot evaluation.

Second, a fixed test dataset that is representative of production workload. Select 2,000 to 5,000 transactions (or entity records, or customers, depending on what you are testing) that reflect the actual distribution of your workflow. If your workflow is 70 per cent domestic transactions and 30 per cent cross-border transactions, the test dataset should have the same distribution. Include transactions that you know should be flagged (true positives), transactions that should not be flagged (true negatives), and a representative sample of edge cases. Run all vendors against the same dataset. This sounds obvious; most institutions do not do it.

Third, a fixed pilot duration and resource commitment. Run the pilot for a defined period (six weeks is typical) with a fixed set of resources (one dedicated resource from the institution, one dedicated resource from the vendor). Do not allow vendors to request extensions or additional resources. The pilot is time-boxed and resource-boxed deliberately to reflect the constraints under which the vendor will operate in production.

Fourth, independent evaluation against the success criterion. Have a second-line or compliance team (not the procurement team that selected the vendors, and ideally not the team that will operate the vendor in production) evaluate the pilot results against the pre-defined success criterion. Score the results on a numerical scale (e.g., entity-matching accuracy from 0 to 100 per cent). Record any failures to meet the success criterion and any differences from the expected behavior. Do not attempt to give partial credit for "close" results; use the criterion as a binary gate.

Fifth, a process for understanding failure. If a vendor does not meet the success criterion, document specifically where it failed. Was it accuracy? Latency? Integration complexity? Cost? Understanding the failure mode is more valuable than the pass/fail result itself, because it guides the next step (either fixing the issue before production deployment or eliminating the vendor).

Grounding This in Operational Constraints

This structure works best when the success criterion maps directly to a constraint that the institution faces. For example:

If the institution's constraint is false-positive rate (staff reviewing too many alerts), then the pilot success criterion should be "the vendor must achieve false-positive rate under X per cent for alert volume Y."

If the constraint is integration complexity (limited engineering capacity), the criterion should be "the vendor must deliver production-ready integration via API without customisation, validated by our infrastructure team."

If the constraint is cost (budget or FTE), the criterion should be "the total cost of ownership (including licensing, integration, and ongoing support) must be under X per annum."

The constraint drives the criterion, and the criterion drives the pilot design. An institution that is budget-constrained should design a pilot that stress-tests cost assumptions. An institution that is staffing-constrained should design a pilot that evaluates integration complexity and support responsiveness, not accuracy metrics.

Stakeholder Implications

For the buyer: A well-designed pilot reduces risk in procurement by eliminating the evaluation phase as a source of uncertainty. If the pilot is structured as a falsifiable experiment and the vendor does not meet the success criterion, that is information; it is not debatable. This simplifies governance sign-off because the second line can review the pilot results and either see that the vendor met the criterion or did not. Poorly designed pilots force the governance team to re-adjudicate procurement decisions based on subjective assessments of pilot quality, which is time-consuming and often contentious.

For the vendor: A well-designed pilot is actually fairer to vendors than a loose pilot is. If the pilot is clear about what constitutes success, vendors can allocate resources to precisely the areas that matter to the buyer and can invest in areas likely to influence the outcome. A vendor that knows the success criterion is "95 per cent entity-matching accuracy on a fixed dataset" will deploy engineering resources accordingly. A vendor that does not know what success looks like will deploy resources generically, hoping to impress across multiple dimensions.

For the second line: A well-designed pilot produces a clear output that the second line can review and ratify. The second line does not need to adjudicate between subjective vendor assessments; it can simply verify that the pilot was run as designed, the data was consistent, and the success criterion was met or not met. This is a mechanical check, not a business judgment call, which is appropriate for second-line governance.

Suggested Next Steps

Before any pilot starts, write the success criterion as a single sentence and have procurement, operations, the second line, and the shortlisted vendors review it. Do not proceed to pilot unless all parties agree the criterion is clear and measurable.

Create the test dataset before the pilot begins, make it available to all shortlisted vendors simultaneously, and announce it in the RFP so vendors can plan resources accordingly. Do not create separate datasets for each vendor.

Assign one dedicated resource from your institution to the pilot (not three people splitting time) and agree with the vendor that they will assign one dedicated resource to the pilot. Most pilot delays come from resource contention.

Schedule a post-pilot review with procurement, operations, and the second line within one week of the pilot end date. Do not attempt to re-negotiate the success criterion in retrospect if a vendor narrowly missed it; if the criterion was clear and falsifiable, use the pilot result as information, not as a starting point for debate.

Sources: EBA outsourcing guidelines, DORA operational resilience requirements, FCA procurement and outsourcing framework, PRA operational resilience expectations, TrustSphere Risk Index, October 2026.