Why most AI governance frameworks fail

I sat down with Anna Guo on the Legal Benchmarks podcast to talk about why so much AI governance manages nothing, and what to do instead. I wrote Singapore's AI Risk Management Guidelines, and the short version of my argument is this: most governance frameworks get stuck on principles and policy drafting, so they turn into an academic exercise. Real control comes from treating AI risk as a system, not a checklist. Here are the points I made, written up.

"Governance" gets stuck on principles

The reason most AI governance frameworks are not very helpful is that they get stuck on principles and policy writing. They become an intellectual exercise. That is why I prefer the term AI risk management. In finance we have managed risk for decades; it is a fundamental discipline, and it is not about vague principles. It is about the concrete question: what systems and processes do we put in place, and what steps do we take, to actually reduce the risk of using something. Principles that tell you to be fair, accountable and transparent sound fine and change nothing on their own. The work is in the detail.

It is a system, not a checklist

Risk management is not a to-do list where you tick a box for a policy here and a reviewer there. You can have a thousand checkmarks that look excellent on paper, and if each one sits in isolation it still does not add up to risk management. The parts have to work as one system. Assessment and testing are part of it, so is explainability, so is human oversight, but they only mean something together.

Start with where your AI is, and how much it matters

Before any of the granular controls, two things have to be in place. First, you have to know where AI is being used - some you build, some you buy, and some arrives inside services you would never expect to contain AI. Second, you cannot apply the same rigour to everything. In a world where AI is becoming ubiquitous, holding every use to the same standard is a losing battle that pulls effort away from what matters. So you assess how material each use is, and you record it in an inventory, even if you only have a few use cases, because the inventory is what lets you apply controls consistently and scale across the organisation. Only once that system exists can you sensibly move to explainability, evaluation and testing, and third-party checks - each calibrated to the use, not applied flat.

The controls are interconnected

Look at the AI governance frameworks around the world and you find the same handful of overlapping areas: explainability, fairness, human oversight, model selection, evaluation and testing, technology and cyber security, monitoring, and change management. They are not separate boxes; they lean on each other. Explainability is a good example. We say AI must be explainable, then stop at "the model has a method that shows how it reached a prediction." That is not the point. The point of explainability is to understand how the system works well enough to evaluate it and to account for a decision to a customer. And if there is no neat explainability method, you are not stuck - from first principles you can still probe behaviour with clear questions, test cases, and red teaming. Explainability serves evaluation.

Human oversight is not a human to blame

Human oversight does not mean dropping a person into the loop. Putting a human in the loop without designing the structure around them is a recipe for disaster, and a little hypocritical - it often just means you have found someone to blame. Real oversight is designing the process so that, where it matters, a capable person can step in and combine their judgement with the AI to get a better result. The person has to be qualified for the check, given enough time to do it, and working inside a process built to actually catch the problem. The recent law-firm cases where fabricated citations slipped through are the lesson: there were policies on paper, including a second review, but a rushed, unsupported human is not a control. In many of those cases a simple rule-based check - a citation whitelist, a retrieval step that verifies the source exists - would have caught more than a tired reviewer did.

Third-party AI: does it fit your context, and do you know when it changes?

Most of the AI a firm uses it did not build, and the providers often have more negotiating power and disclose less than you would like. Two things to hold on to. First, whatever the vendor tested or benchmarked was built to generalise across many customers - so the question is whether it fits your context, your workflows, your task. Their benchmark is not your evidence; you may need to test it yourself. Second, do not assume a change means only the model changed. Every model today sits inside a system, a harness around it, and a small change there - even with the same model underneath - can shift your workflow in ways you did not expect. You need a way to know when a provider changes the model or the system, because when they do, things change in practice for you.

Risk management is not a brake on innovation

The worry that governance slows you down has it backwards. Everyone knows proof-of-concept hell: impressive pilots that never ship, or ship and go unused, or ship and then produce an incident. Evaluation, testing, and understanding where a use case works and where its behaviour breaks down are prerequisites for AI that is actually effective and useful. An incident sets you back far further than doing the work up front - in law and finance the reputational cost is enormous. Done properly, risk management is not the auditor ticking boxes. It is how you know the thing will work the way you need before you rely on it.

Watch the full conversation

The full discussion, including the legal profession's role in AI risk and where regulation is heading, is on the Legal Benchmarks podcast, hosted by Anna Guo.

Work with me

This is the thesis behind how I teach and advise: AI risk management as a working system, grounded in the AIRG and read against the other frameworks. See the courses and workshops, or get in touch.