"Governed AI" gets used a lot right now, often to describe very different things. For some vendors it means a disclaimer in the terms of service. For others it means a human somewhere in the organisation is nominally responsible for the AI feature if something goes wrong. Neither of those is governance — they're liability management dressed up as a product feature.

In a sector like health and social care, where an AI system's output can influence decisions about a person's safety, wellbeing or regulatory standing, governance has to mean something more specific: architectural constraints that exist whether or not anyone remembers to check them.

Three things governance has to guarantee

For AI to be trustworthy in a regulated environment, it needs to satisfy three properties at the architecture level — not as policy, not as training, but as constraints the system cannot violate even if someone wanted it to.

It has to be explainable. If an AI system flags a risk or recommends an action, the person reading that output needs to see the evidence behind it in plain language. A number with no reasoning attached — a "risk score" of 74, say — is not explainable. It's an oracle, and oracles don't hold up in front of a regulator, a family member, or a board.

It has to be permission-scoped. An AI feature that can surface information a user isn't otherwise entitled to see has broken the organisation's access model, regardless of how useful the output is. Governance means the AI layer inherits the same permission boundaries as everything else in the platform — it doesn't get a side door.

It has to defer to a human at the right threshold. Not every AI output needs sign-off — that would make the system useless. But anything that crosses into safeguarding, financial commitment, or regulatory risk should stop and wait for a person before it takes effect. Deciding where that threshold sits is a genuine design decision, and it has to be made deliberately, not left to whatever the underlying model happens to do by default.

Why this has to be decided early

The uncomfortable truth is that these properties are very difficult to retrofit. If an AI feature was built first and governance was added afterward, the explainability, the permission scoping and the approval thresholds tend to be shaped around what the feature already does — rather than the feature being shaped around what governance requires.

That's why, at iCura, AI governance isn't a layer we added to a set of AI features. It was a precondition for building any of them: no AI capability ships until it can explain its own reasoning, respects the same access boundaries as the rest of the platform, and has a defined human checkpoint for anything above a low-risk threshold.

What this looks like day to day

In practice, this means a registered manager using iCura's AI Decision Support never sees an unexplained recommendation. A flagged compliance gap comes with the specific record, date and standard that triggered it. A staffing risk comes with the reasoning behind it — not just an alert, but the "why."

And when the system identifies something serious enough to matter — a safeguarding concern, a significant compliance gap, a change with financial consequences — it doesn't act. It routes to a person, with the context they need to make the actual decision.

That's the difference between an AI feature and a governed AI system: one is impressive in a demo. The other is something you can defend in front of the people whose job it is to ask hard questions.