AI bias and fairness in finance

What it is

AI bias is a model being systematically worse for one group than another. The reason it is dangerous is that it hides in the average: overall accuracy can look healthy while one group is served badly. It helps to separate unequal outcomes from unequal errors, and to be clear that it is rarely deliberate. It arrives through the data and the design, through what you might call six doors: historical, representation and measurement bias in the data, and aggregation, evaluation and drift bias in how the model is built and watched. And there is a harder point as well: the common definitions of fairness cannot all hold at once, so fairness is a decision the firm makes and defends, not a number it measures.

Why it matters in finance

Credit, pricing, claims and hiring are the obvious places. Treating similar customers differently in ways that are unfair, or unlawful, is a conduct risk before it is a model problem, and in regulated decisions it carries direct legal exposure. The stock example is the hiring tool that learned to prefer one group from historical data and kept applying that preference.

What catches it

Choose a fairness definition that fits the use and defend the choice. Run disaggregated evaluation, testing each group rather than the average. Add data controls, an explainability check for proxy variables standing in for a protected attribute, human oversight with the authority to stop the system, and monitoring, because a system that was fair at launch does not stay fair. The joins that break most often are fairness claimed without disaggregated evaluation, a data scrub with no proxy check, and fairness tested at launch but never again.

What to ask

Which definition of fairness is being used, and who chose it? Was the system evaluated group by group, not just overall? Was there a check for proxies? Does oversight actually have the authority to halt it? Is fairness re-checked in production?

Related on this site: testing, evaluation and assurance, data ethics, explainability and opacity, and the guide for risk and control functions.

In an ebook: AI Risk Management for Risk and Compliance.