Quaintitative

For regulators and supervisors

AI risk management for regulators and supervisors

You do not assess the model. You assess whether the firm has managed it.

A supervisor coming to AI early often worries which framework to base a regime on, and whether they have already fallen behind. It is the wrong worry. Lay the approaches on a line and they spread out along one axis: binding law like the EU AI Act at one end, voluntary standards and principles at the other, and principles-based, sector-specific regimes like the MAS AIRG in between. What moves along that line is force. What does not move much is what they ask: someone accountable, a firm that knows where its AI is and rates each use, heavier controls on the riskier uses across the lifecycle, and the capability to run it all. That architecture is, in substance, shared by the AIRG, the NIST four functions, ISO 42001 and the EU AI Act's obligations for high-risk systems.

Look at that list from your seat and it is supervision you already do. A risk-management system that finds and rates exposures is what you hold firms to for their models. Lifecycle controls with independent validation is model risk supervision. Governing what a firm buys is your outsourcing rules. Watching live systems and managing change is technology and operational risk supervision. The frameworks are not handing you a new discipline; they are pointing at the supervision you already do and asking you to extend it to AI. The convergence is a gift: anchor on the instrument with standing in your market, and know the work maps substantially onto the others.

A supervisor does not assess a model; you assess whether a firm has managed it. Under the MAS AI Risk Management Guidelines (AIRG), that means judging the adequacy of the firm's own AI risk management: whether identification works as a net rather than a list, so that embedded and shadow AI are caught; whether materiality is assessed honestly and drives the depth of control; whether the firm retains accountability for the AI it buys, including concentration on a few providers; and whether accountability is real and demonstrable. The firm runs the controls. Your job is to test whether it can show they work.

The practical test is what you find when you ask for evidence: an inventory, a materiality rationale, test results, an escalation record, produced on request, or a set of good intentions.

A few questions I would start with

These come from years of supervising risk management, writing the AIRG, and leading the thematic review of how banks actually manage AI model risk. For each, the question I tend to hear from supervisors, and how I think about it.

Require one approach, not one shape

"Should we tell firms what operating model to run: a central AI function, hub-and-spoke, or a capability in every unit?"

No. A shape that fits one firm fails the next, and the day you endorse a structure you own it. Require consistency instead and leave the shape to the firm. Consistency is also what you can verify from outside: if one unit calls a model AI and another does not, the firm-wide numbers are fiction, and if a 'high' on one desk is a 'medium' on the next, the risk appetite cannot be relied on. Focus on whether one approach governs AI risk the same way across the whole firm, not which box on the org chart it sits in.

Test the net, then go to the low-rated

"How do we know a firm has found all of its AI?"

You cannot verify completeness from outside, and chasing it will exhaust you. But you can test the net: ask how AI arriving through a licensed tool's new feature, a vendor's update, or a staff chatbot reaches the inventory, and when the firm last caught one it had missed. Then spend your hours against your instinct, on the systems rated low rather than the showcase high-rated ones, because the low rating is the cheapest way for a firm to make risk disappear. Pick the sample yourself. Focus on whether the firm has a good net to catch AI as it enters, and whether each low rating would survive a second look.

Ask what the inventory does

"The firm has an AI inventory and showed it to us. Isn't that the box ticked?"

Most inventories are compiled for you: accurate the week before the inspection, stale the rest of the year, read by nobody. That is a list, and a list governs nothing. Ask what only a working inventory answers quickly: which systems depend on this one vendor, what changed last quarter, what was retired this year. A firm with a working inventory answers in minutes; a firm with a list convenes a working group. Two questions tell you which it has: does anything depend on the inventory, and does anything write back to it?

Test to the task, not the benchmark

"The firm shows us benchmark scores and a working demo. Is that evidence?"

No. A benchmark is someone else's exam, and the questions may have leaked into the training data. A demo is the one path the firm chose to walk you down. You do not need to re-run anything or read the model's weights. Make one request: show me what a good and a bad answer look like for this task, the threshold you set, and the results against it, on your own data including the cases where it did badly. A firm that has done the work answers in a sentence and a table; one that has not sends a deck about the model. Focus on whether the firm can show what good enough means for the task, in its context.

Look for a system, not a checklist

"The firm handed us a control mapping that covers our guidelines row by row. Is it compliant?"

The mapping was produced for you, which is why it proves little. Controls work as a system, not a list: identification feeds the inventory, the inventory holds the ratings, the rating sets the intensity of every lifecycle control, monitoring feeds back, and a change starts the cycle again. You cannot test a system by touring its policies, so trace a thread: pick one material system and follow it end to end, from how it was identified to how it was rated, what controls that pulled, what monitoring has done since, and what happened when it changed. One thread followed honestly tells you more than the whole mapping. Focus on whether the parts reinforce each other, not how many rows the mapping has.

Ask where the firm decided fast is safe

"Firms ship AI faster than our inspection cycle. Are we just always behind?"

Your inspection cycle will never match their release cycle, and it does not need to. The check is whether the firm decided, in advance, where it can move fast: expedited paths for genuinely minor, pre-agreed changes, and clear boundaries for pilots, with limits on time, users and production data. A firm that is slow everywhere is not careful, it is undiscriminating. The same discipline applies to your own regime: if every model change needs your nod, you have made yourself the bottleneck and the owner of every outcome you nodded through. Focus on whether the firm has deliberately decided where it can move fast, not whether every review is fast.

These are where I start. The fuller treatment, including the two choices the frameworks leave you, how binding and broad your regime is and how you actually supervise against it, and where firms hide risk, is in the book: a short primer, and the full AI Risk Management for Regulators and Supervisors.

Where to start

I developed the AIRG while leading AI risk supervision at MAS, and now advise regulators and supervisors independently. I am publishing more on AI risk management; subscribe to get it, and future updates on the AIRG.

Subscribe for updates