AI data leakage risk in finance
When a model gives away what it should not, training data, the personal data inside it, or one customer's information to another. One risk from the wider taxonomy.
What it is
A model remembers. The leak is any path from that memory, or from the run-time prompt, back out to the wrong person. There are a few routes: the model reproducing something from its training data, an attacker inferring who was in the training set, deliberate extraction of records, personal data appearing in an ordinary output, and context leak, where one user's data surfaces in another's session. A useful way to see it is that the leak is often a correct output sent to the wrong recipient. The line that organises the whole problem is simple: what you never put in cannot leak.
Why it matters in finance
Finance runs on data that must not move: account records, KYC files, and anything sensitive placed in a prompt or sent to a third-party model. A leak here is not an abstract privacy point. It is a customer's records reaching someone who should not see them, or confidential data leaving the firm through a model, a log or a vendor, often with no obvious trace.
What catches it
Data management and security are the primary controls, and they work by reducing what is exposed: minimise what goes in, redact what the task does not need, and isolate so one session or tenant cannot reach another's data. Around that sit extraction red-team testing, monitoring for personal data in outputs, vendor egress terms in the contract, human oversight, and a materiality judgement about how sensitive the data is. Differential privacy is a training-time option where it fits. A useful test: if someone had an afternoon and access to this system, what could they draw out of it?
What to ask
What sensitive data does this system hold or receive? What stops it leaving through a prompt, an output or a log? Could one customer's data surface in another's session? What did the vendor commit to on where data goes and how long it is kept?
Related on this site: AI security risks, prompt injection risk, data management, governing third-party AI, and the guide for risk and control functions.
In an ebook: Agentic and Generative AI for Finance.
One risk from the wider set. See the full AI risks in finance taxonomy.