Runtime governance for agentic AI

Traditional model governance assumes a model you can test, approve, and review on a schedule. An agent breaks that assumption, because the thing that carries the risk is the run itself: the actions it takes, in what order, using which tools. You can approve the agent on Monday and still not know what it will do on Thursday. So governance has to move to runtime.

The approach my co-authors and I set out in Scalable Runtime Governance for Agentic AI in Financial Services starts by not treating the whole agent as one opaque thing. You decompose its workflow into a few reusable capabilities, each a bounded piece of work with explicit authority (what it is allowed to do), explicit limits (what it is not), and the evidence it must produce. You validate those capabilities once and reuse them across workflows, which is what makes the approach scale rather than re-reviewing every agent from scratch.

Then you govern the running agent. In short, that means:

• Telemetry. Capture what the agent is actually doing, its reasoning steps, the tools it calls, and the actions it takes, so there is a record to check against.
• Continuous authorisation. Authorise actions as they happen against current policy, rather than granting one blanket approval up front.
• Policy conformance. Check each action against the rules in the moment, and stop the ones that breach them.
• Drift monitoring. Watch for the agent's behaviour wandering from what was validated, not only its outputs changing.
• Tiered containment. Have graduated limits, and a kill switch for high-materiality agents, that can be triggered and are tested regularly.

This is not a fringe view. The MAS AIRG already points the same way: it expects monitoring to extend to an agent's reasoning, actions and tools where relevant, and kill switches for high-materiality AI that are tested. Runtime governance is how you meet that in practice.

The paper carries the full treatment. For the wider picture, see agentic AI risk management, governing agentic AI, and the final AIRG.

Subscribe for updates