Prompt injection risk in finance
When instructions hidden in a model's input steer it off task, and why that only becomes dangerous once the system can act. One risk from the wider taxonomy.
What it is
Prompt injection is when instructions hidden in the content a model reads steer it into doing something it should not. It comes in two forms. Direct injection is a user typing a hostile instruction. Indirect injection is the more serious one: the instruction is planted in a document, a web page or an email that the model retrieves and treats as part of its task. People call it the new SQL injection, and the comparison holds, because the root cause is the same: instructions and data travel in one channel, so the model cannot reliably tell a command from content. There is no clean fix. You reduce the risk; you do not remove it.
Why it matters in finance
On its own, a manipulated answer is just a bad answer. The risk turns real when the model has tools, because then text becomes an action. A model that can send an email, move a file, update a record or call an interface can be made to do those things by text it reads. The point where hostile text becomes an action is the point to worry about, and permissions matter more than the prompt.
What catches it
Security is the primary control, built on least privilege: the model can touch only what the task needs, so the worst injected instruction still cannot reach anything irreversible. Around that sit red-team testing that probes through everything the model reads, monitoring for guardrail hits, vendor and supply-chain checks, and human approval for high-impact actions. Good enough is a plain test: the attack you missed cannot reach something you cannot undo.
What to ask
What can this system actually do if an instruction in its input tells it to? What are its permissions, and who set them? Has anyone tried to steer it through a document it retrieves, not just through what a user types? What is logged when a guardrail fires, and who looks at it?
Related on this site: AI security risks, data leakage risk, governing agentic AI, making AI safe in practice, and the guide for risk and control functions.
In an ebook: Agentic and Generative AI for Finance.
One risk from the wider set. See the full AI risks in finance taxonomy.