AI explainability and opacity risk
When a decision cannot be explained to the customer it affected or the regulator who asks, and why a reasoning-style trace is not proof. One risk from the wider taxonomy.
What it is
Opacity is the failure where a decision cannot be explained. A useful way to see it is three kinds of visibility. The glass box, where you can read how the model decides. The frosted box, where you can only approximate it. And the mirror, where the system produces a reasoning-style account that reads convincingly but need not reflect how the output was actually produced. The mistake is treating the third as the first. Accurate is not the same as explainable, and a plausible trace is not evidence of the real basis for a decision.
Why it matters in finance
You owe an explanation to a customer who was declined and to a regulator who asks why. Opacity is usually not forced on the firm; it is an unframed choice. Complexity gets taken on without anyone asking whether a simpler model would do the job, and then it is carried for the life of the system. And dashboards get sold as explanations, so a firm can believe it has transparency it does not actually have.
What catches it
Explainability is the primary control, but with two conditions: the explanation has to be faithful, reflecting the real basis of the decision, and fit for its audience, readable by a customer and by a regulator. Alongside it sit model selection (could a simpler, more transparent model do the job), human oversight (does a person actually read the explanation and act on it), and documentation (is the decision on the record and reconstructable next year). The join that breaks most often is an explanation that no one ever tests for faithfulness, which leaves every other control depending on an account no one has checked.
What to ask
Could a simpler model have done this? Is the explanation faithful, or just plausible? Can a customer and a regulator both read it? Does a human actually use it? And could you reconstruct the decision a year from now?
Related on this site: testing, evaluation and assurance, over-reliance and automation bias, reproducibility and traceability, governance documents, and the guide for risk and control functions.
In an ebook: AI Risk Management for Risk and Compliance.
One risk from the wider set. See the full AI risks in finance taxonomy.