Can you eliminate AI hallucination?

Not entirely, because it comes from how the models work. You reduce it: ground the model in approved sources you supply, limit what it can reach, and design it to decline when it is unsure rather than answer anyway. Then you measure whether answers are supported by their sources and review the outputs of material uses.

AI Hallucination Risk in Finance

Q: Does AI hallucination matter in finance?

Yes. In finance the output can affect customers, records and money. A confident, well-written answer that carries the wrong fee, an invented contract clause or a fabricated citation can be acted on before anyone checks it. So you need evidence the answer is right for the task, not just that it reads well.

Q: How do you test for AI hallucination?

Build a test set from real work, with written criteria for what a good answer must contain. Measure whether each answer is actually supported by the source it points to, how often the system answers when it should have declined, and how often a person had to correct it. Re-run the tests after any change, and keep the worst cases. A citation is not proof, so inspect the retrieved source when an answer is wrong.

What it is

A language model generates the most likely next piece of text. It does not check what it produces against an authoritative record. So the same fluent style can carry a correct figure and an invented one, and the invented one is what people call a hallucination. The output reads with the same confidence either way, which is what makes it dangerous: nothing in the text shows which is which.

It helps to be precise about the cause, because it shapes the control. The model is not lying and it is not malfunctioning. It is doing exactly what it was built to do, which is produce plausible text. Hallucination is a property of how these models work, not an occasional bug, so the aim is to manage it down to an acceptable level for the use, not to switch it off.

Why it matters in finance

In finance the output can affect customers, records and money, and it is often acted on quickly. A relationship manager pastes a summary into a client note. A colleague copies a figure into a report. An answer about a fee, a loan covenant or a product feature is treated as settled because it came back clean and well written. A hallucinated fee, an invented contract clause or a fabricated citation causes harm when it is acted on before anyone checks it.

The risk rises with two things in particular: how far the output reaches, and how much a person relies on it without a check. Those are the same factors the materiality rating already uses, which is why hallucination is a risk to be rated and controlled, not a novelty to be feared.

What the AIRG expects

The MAS AI Risk Management Guidelines are risk-based and proportionate, so the depth of control follows the materiality of the use. Hallucination is one of the generative-AI failure modes the guidelines expect a firm to account for, alongside manipulation through the input and leakage of sensitive data. Rate the use on what it can affect, including how far the output reaches and how much people rely on it, and let the controls scale with that rating. For a material use, testing has to cover behaviour and not just accuracy, and an independent review before deployment is carried out by someone not involved in building it.

What reduces it in practice

Ground it in your own sources. A language model works, for one request, only from what it is given. The strongest control is to make it answer from approved documents you supply, and to show which part of the source it used. For a question about a fee or a covenant, the system should pull the relevant document and answer from it, rather than from whatever it absorbed in training. Retrieval helps, but it is not a guarantee: a strong model can still misread a good source, and a citation is not proof, so the retrieved material has to be inspectable when an answer is wrong.

Limit what it can reach, and keep records. Give the system access only to what the task needs. Keep a record of what was asked, what was retrieved and what was produced, so a wrong answer can be investigated later rather than argued about.

Let it decline. A system that always answers will always, sometimes, answer wrongly. Design it to say it does not know, or to route to a person, when the source does not support an answer. For a material use, abstaining is usually better than a confident guess.

Put a person where the stakes are. Human review scaled to materiality, from a sample for a minor use to review of every output for a material one. The point is that the reviewer can actually catch a wrong answer, which means they need the source to check it against.

How you know it is working, and what to ask

A control you cannot measure cannot be relied on. Build a test set from real work, including the hard cases: incomplete requests, out-of-date or conflicting sources, and cases where the system should ask for help rather than answer. For each case, write down what a good answer must contain. Then measure three things: whether an answer is actually supported by the source it points to, how often the system answers when it should have declined, and how often a person had to correct it. Re-run the tests after any change, and keep the worst cases.

If you sit in the second or third line, the questions follow from that. How was this use rated, and does the rating hold up given what the output can affect? What was the system tested on, and did the test set come from real work rather than a clean sample? How is groundedness measured, and what is the abstention behaviour when the source does not support an answer? What does monitoring show after go-live, and what happens when a person keeps having to correct the same kind of answer? A public benchmark can help shortlist a model, but the decision to go live needs evidence from your own tasks.

In an ebook: Agentic and Generative AI for Finance.