AI over-reliance and automation bias

What it is

Over-reliance, or automation bias, is the tendency to defer to a system because it is confident, fluent and convenient, and to stop applying the judgement the control assumed a person would apply. The person meant to check the system stops checking it. A clean dashboard or a well-written answer makes it worse, because a confident presentation looks like reliability, and it is sharpest where the output is hard to check and easy to accept.

Why it matters in finance

A great many AI controls rest on a person reviewing, challenging or overriding the system. That is the whole premise of human oversight. If the reviewer has stopped checking, the control no longer works, even though it is still there in the design. The exposure grows as the output looks more authoritative and as volume rises, so the busiest, highest-throughput processes are where it is greatest.

What catches it

Design oversight so the reviewer can actually catch a wrong answer, which usually means giving them the source to check against, not just the answer to approve. Match oversight to materiality rather than asking for blanket sign-off that no one has time to do properly. Measure the override rate and look hard when it falls towards zero, because near-perfect agreement usually means no one is really checking. And do not present an unreliable output as if it were settled fact.

What to ask

Where does a control depend on a person checking the AI? Can that person actually verify the output, or only accept it? How often do they override it, and would anyone notice if they stopped? Is the output shown with its limits, or presented as certainty?

Related on this site: explainability and opacity, AI risk management for boards, monitoring, and the guide for risk and control functions.

In an ebook: AI Risk Management for Risk and Compliance.