How to evaluate an AI portfolio review pilot on your own documents

1. Write the pass criteria before the pilot starts

Put the pass criteria in the statement of work for the pilot, before the vendor sees a single document. That way any decision to go further rests on agreed scores. Each criterion names the measure, the threshold your team sets, the person who scores it, and the decision the result triggers.

A workable set for a portfolio review covers five measures:

• Traceability. The share of figures in the review pack whose evidence opens to the exact page, table, or cell for the right period and shows the same value.
• Rules. The share of rule cases in the test set the pilot handles correctly, with the right rule named whenever it refuses.
• Change and gaps. Whether every changed KPI definition is flagged and every missing fact is named.
• Review time. Hours from receipt of company reports to sign-off, measured against today's baseline (Step 7).
• Corrections. Whether a correction made in one period holds in the next.

Then write down what each result means. A pass on all five opens the discussion about going further: more companies, more periods, or a second decision. A failure on traceability or rules ends the pilot, whatever the review time shows. Mixed results get a named fix and a date for a rerun on the same test set. For what sets the cost of a deployment, see how pricing works.

2. Build a test set from periods where you already know the answer

Pick two or three recent review periods your team has signed off, and record the correct output for each before the pilot begins. Use the files exactly as they arrived: scanned management accounts, board decks, lender covenant certificates, Excel workbooks with hidden tabs and merged cells, and the exports from your portfolio monitoring platform. Cleaning them first removes the work you're paying the pilot to do.

Most of the set should be ordinary lookups. The hard cases separate one tool from another, so give each hard case a known answer and a place in the score.

Take a made-up portfolio company, Brennick Logistics, in a made-up fund. Their Q2 management accounts report EBITDA of 4.1 million euros on page 6, while the covenant certificate in the lender pack reports 4.6 million. The certificate adds back 0.5 million of one-off restructuring costs, and the fund's own EBITDA definition disallows that add-back. The finance team's KPI workbook carries a third figure, 4.3 million, keyed in before a late adjustment to the management accounts. In Q3, Brennick starts counting multi-year service contracts as recurring revenue, and the bank statement for one subsidiary is missing on the review date. The fund's policy sends any valuation input that moves more than 10% to the deal partner.

Your team records the expected outcome for each case before the pilot sees the files:

The four outcomes in the middle column come from Saphenia. They work as a scoring frame for any tool, since every output in the test set should land in exactly one of them. An output that lands in none of them, such as a confident figure with no source, counts as a fail.

3. Write your reporting rules down and test that they hold

A tool reflects your reporting rules when it refuses an output that would break one and names the rule. To test that, first write the rules for the review in one place, starting with which source outranks which: audited accounts over management accounts, management accounts over the KPI workbook. Add how the fund defines EBITDA and recurring revenue, which thresholds send a figure to a person, and which figures always need a partner's sign-off. Then build test cases where the easy answer breaks a rule, like the Brennick ratio above.

A model can ignore a rule written into its prompt, so test whether every output in the set obeys your rules. Partway through the pilot, change one rule, for example by raising an escalation threshold, and rerun the same period. The outputs should follow the new rule, and the record should show which rule version applied. Finally, correct one figure in a signed-off period and run the next period to check that the correction carries forward. For how that correction gets recorded, see who signs off a portfolio figure.

4. Test facts that change between periods

Include at least two KPIs whose definitions changed between periods, and expect the pilot to flag each change with both definitions dated. A pass shows the figure under the definition your fund reports on, states when the company's definition changed, and keeps the earlier period comparable. A fail is a quarter-over-quarter growth figure that mixes the two definitions with no flag.

Add a superseded document as well. Put last year's management accounts in the folder beside this year's, and check that every figure cites the version for the period under review. For more on changed definitions, see changed KPI definitions and missing facts.

5. Include figures that sit in separate systems

Put at least two figures in the test set that have to be assembled from more than one system, and at least two that the sources report differently. Net debt is a good candidate: the drawn balance comes from the lender's facility statement, cash from the bank statements, and lease liabilities from your portfolio monitoring platform. A pass shows each input with its own evidence, the calculation, and the date of each source.

Where two sources disagree, the pilot should show both figures with their dates and sources, then apply your source hierarchy. If your rules send that conflict to a reviewer, the output is reserved for a human and says which figures conflict. For that case in detail, see when portfolio reports disagree.

6. Trace every figure in the test set to its source

Before you trust source citations, open every one in the test set. Check the document version, the period, and the exact page, table, or cell, and confirm the value there matches the value in the review. A citation that opens the right file on the wrong page counts as a fail.

A review pack that links each figure to a whole file leaves the checking to your team, so ask for the exact location behind every figure, plus the rule version that produced it. TechTarget's James Alan Miller made a related point about agent logs in his feature on decision trails: "A dashboard can show that an agent completed a step. It cannot prove that the agent should have completed that step."

When a pilot's source attribution looks weak, check these points:

• Derived figures (margins, growth, net debt to EBITDA) show each input with its own evidence and the calculation.
• Citations point to the version of the document for the period under review.
• A figure absent from the documents comes back as a named gap.
• The record shows which rule version applied to each figure.

Then find where attribution breaks, because each kind of error needs a different fix. Extraction errors misread a value or point to the wrong page, calculation errors drop inputs, and versioning errors cite an old document. The criteria from Step 1 decide whether the vendor gets a rerun.

7. Measure review time against today

To judge review effort on your own numbers, time one recent review period before the pilot begins: hours from receipt of company reports to sign-off, per company, including the time spent chasing and checking figures. Then measure the same span during the pilot, with checking time counted in full. If reviewers still re-check every figure the tool drafts, the review takes as long as it did before.

Base the production decision on your own figures: hours per company before and during the pilot, the corrections reviewers made, and how many outputs landed in each of the four outcomes. A vendor's accuracy percentage describes the vendor's test set, so record your own on the cases from Step 2. Track corrections across periods as well, since reviewers should make fewer of them each period as approved corrections carry forward.

Where Saphenia fits

Saphenia runs a portfolio company review as a decision workflow, one workstream at a time. Saphenia reads your documents and systems and writes down what the review follows: what your terms mean, which source wins when two disagree, and where the thresholds sit. Your reviewers then correct the agent's work, and every correction an expert approves becomes a rule for every later review.

Saphenia checks every claim against your rules before the answer goes out, and each output lands in one of the four outcomes from Step 2. Each proof carries the facts, the rule, its version, and the date, so a reviewer follows any figure back to its source.

At Saphenia, we start every engagement with a live agent on one real decision, and that agent is what your test set scores. The decision logic and everything built from your data stay yours, in the environment you choose. For where it runs and who owns the data, see security and hosting.