AI reproducibility and traceability risk
When you cannot reconstruct how an output was produced, you cannot investigate it, defend it or fix it. One risk from the wider taxonomy.
What it is
This is the risk that you cannot reconstruct how a given output was produced, so you cannot investigate it, defend it or fix it. Predictive models already need this discipline; generative systems make it harder. The same input can give different outputs, because these systems are not deterministic, and the result depends on the model version, the prompt, the retrieved context and the underlying data, any of which may have changed since the output was produced. Without a record, there is no way to re-examine how an output was produced.
Why it matters in finance
When a decision is challenged, by a customer, an auditor or a regulator, the firm has to show how it was made. If there is no record of the model version, the inputs and the sources it drew on, an incident cannot be investigated and a control cannot be shown to have worked. A control you cannot evidence is, for practical purposes, a control you do not have.
What catches it
Logging detailed enough to reconstruct an output: what was asked, what was retrieved, and what was produced. Version control over models, prompts and data, so you know which versions were in play. A golden set and fixed test conditions, so results can be compared over time rather than being lost. And documentation on the record, so the decision can be reconstructed next year, not just this week. These are the traceability side of the monitoring and evaluation controls, rather than a separate programme.
What to ask
Could you reconstruct how this output was produced, months after the fact? Is the model version, the prompt and the retrieved source actually recorded? Can you reproduce a result closely enough to investigate a complaint? And where is any of this documented?
Related on this site: monitoring, explainability and opacity, model drift, governance documents, and the guide for risk and control functions.
In an ebook: AI Risk Management for Risk and Compliance.
One risk from the wider set. See the full AI risks in finance taxonomy.