briefings

Your AI Governance Will Fail in the Org Chart, Not the Architecture

The recognition

Gartner projects that task-specific AI agents will be embedded in 40 percent of enterprise applications by the end of 2026, up from under 5 percent a year earlier. Deloitte finds that only 21 percent of organisations have a mature governance model for agentic AI. The deployment curve and the governance curve are diverging, and most organisations are trying to close the gap with engineering.

That work is necessary, and it is largely solvable. It is also not where the failure happens. A technically sound AI governance architecture can be fully built and still fail in production, because the part of the organisation it depends on has been deliberately removed over the last decade. The failure is structural, and it sits in the operating model, not the technology.

The mistake hiding inside “human-in-the-loop”

The default response to AI risk is to add more human oversight. More approvals, more checkpoints, more humans in the loop. This conflates two functions that have nothing in common except the word “human”.

The first is accountability anchoring: a human is formally attached to a consequential decision so responsibility can be assigned. It is threshold-triggered and rule-based. The second is diagnostic judgement: a monitoring layer surfaces an anomaly it cannot classify, and the question is no longer “approve this decision” but “what kind of problem is this, and what is the correct response”. Is it a data shift requiring retraining, an adversarial attack, a governance envelope that no longer fits, or a model flaw visible only at scale? Route it wrong and the intervention makes the problem worse.

These are different functions requiring different people and different design. The debate treats them as one. That conflation is why so many governance designs are simultaneously too slow to be operational and too shallow to satisfy accountability.

The layer you already deleted

Diagnostic judgement has a precise organisational home: the middle layer. The function that holds pattern recognition across operations, reads anomalies against intent, and routes the right intervention to the right place.

Organisations have spent more than a decade hollowing out exactly this layer in the name of flat hierarchies and efficiency. The cost was hidden until now. When a monitoring agent surfaces an anomaly, it has to escalate to someone. If the diagnostic layer is gone, it escalates to the operational layer, which lacks the context to diagnose, or to the strategic layer, which lacks operational proximity. Neither can read it correctly. The escalation path leads nowhere, and the governance architecture becomes decorative. You can pass every technical audit and still have governance that does not work, because there is no one whose job it is to interpret what the machine flags.

What this actually requires

The fix is not more technology and not a committee. It is a named, accountable role and a restored layer.

Every consequential AI system in production needs one Model Owner: a named individual, not a shared function, accountable for a specific model from deployment to decommissioning. The competence sits between a data owner and an ML engineer, enough diagnostic literacy to judge whether specialists are routing an anomaly correctly without personally performing the diagnosis. The analogy is a GP, not a radiologist. There is precedent: regulated organisations already created the internal data-owner role to discharge GDPR obligations. Extending that pattern to model governance is a design problem, not a category problem.

The structural read

This is not really a story about AI. AI is the acute symptom. The deeper condition is an operating model built for stable, predictable environments now meeting one defined by uncertainty, emergence, and adversarial pressure. This is the central argument of The Kinetic Enterprise: the industrial-era model concentrated authority at the top, standardised execution at the bottom, and treated the middle as cost to be optimised away. That worked when the environment was predictable. It is structurally incompatible with environments where harmful outcomes emerge from the interaction of individually well-governed parts, faster than central oversight can follow.

The functions were never wrong. The model was never updated.

Where this leaves the executive

If you are accountable for AI in a regulated enterprise, the question is not “is our architecture sound”. It is narrower: when our monitoring flags something it cannot classify, who interprets it, do they have the authority and the literacy to act, and does that role exist on our org chart today? If the answer is unclear, the gap is in the operating model, and no amount of engineering will close it.

If that question does not have a clean answer in your organisation, that is the conversation to have. Start it here.