Clinical governance of AI needs to be anchored in clinical leadership and embedded from the outset, “with real authority – not just advisory input,” according to leading digital health expert Nirvana Luckraj.

Dr Luckraj, director of Nirvai Global Health Advisory, is speaking at next month’s HealthTechX Asia conference in Singapore on the topic of “The Invisible Infrastructure: Why Clinical Governance Makes or Breaks Health AI.”

Dr Luckraj told Pulse+IT the most consistent gap in clinical AI deployments was a lack of clear ownership. 

“AI is often implemented as a digital product rather than a clinical intervention, which diffuses accountability for safety,” Dr Luckraj said.

“Responsibility is often split across technical, operational and clinical leaders. When something goes wrong, ambiguity around who is ultimately responsible becomes a clinical risk in itself.”

She said another gap was “what happens after go-live.”

“Organisations invest in upfront validation and then move on. AI systems don’t stay static and neither do the populations they serve. If you’re not monitoring real-world performance in your clinical environment with your users, you’re operating without meaningful oversight,” Dr Luckraj said, speaking from her experience in the digital health sector – including her former role as chief medical officer at Healthdirect Australia.

“At the national virtual care service I led as chief medical officer, where we implemented AI-enabled triage at scale, one of the most valuable governance mechanisms was the clinician override feedback loop: every time a clinician disagreed with the AI’s recommendation, we recorded it, reviewed it and looked for patterns. That’s how the system improved.”

TRUST FACTOR

Dr Luckraj also pointed to “the cultural layer,” which she said was “almost always underestimated.”

“Even when the technology performs as designed, adoption depends on whether clinicians trust it enough to use it in real workflows. That trust is hard to earn when a system offers recommendations but can’t consistently explain its reasoning in terms clinicians can stand behind.

“The ‘black box’ effect is real, but the issue is not just technical explainability. It’s whether the system can justify its outputs in a way clinicians are willing to stand behind. That is a governance problem as much as a technical one.” 

Dr Luckraj said clinical governance for AI needed to be anchored in clinical leadership, “with real authority – not just advisory input and embedded from the outset.”

She noted that clinicians owned the safety and appropriateness of clinical decision-making. 

“Digital teams may own implementation and integration. Vendors are responsible for the performance and integrity of their models. At a system level, there must be explicit ownership of how these responsibilities interact, otherwise risk falls between the gaps, and no one holds the full clinical risk picture.”

Dr Luckraj said in her advisory work across health systems and digital health organisations, she consistently saw “fragmentation in how these responsibilities are assigned.”

“Clinicians are often brought in too late to meaningfully influence procurement or design. The result is a set of partial accountabilities with no one holding the full clinical risk picture. That gap often becomes visible when something goes wrong.”

RAPID EXPANSION

Questioned about rapid AI adoption and whether governance frameworks were keeping pace, Dr Luckraj warned “What we’re living through is the rapid expansion of AI deployment without an equivalent expansion of clinical assurance capability.”

“This creates a systemic risk – not just isolated system failures, but clinical variability and bias being embedded at scale, particularly when local adaptation is poorly controlled, or when updates are pushed frequently without proportional clinical review.”

She said across the organisations she was currently working with, governance frameworks often existed, but “may be applied retrospectively rather than designed in from the outset – or treated as a compliance exercise.”

Dr Luckraj said the regulatory environment was tightening, “but regulation alone doesn’t build a governance culture.”

“That culture has to come from within organisations with a genuine understanding that governance isn’t a function owned by one team, but a shared responsibility embedded in how the work gets done. Governance that lives only in a policy document doesn’t protect patients.”

GOOD GOVERNANCE

She said good governance was visible in how clinicians interact with the system every day. 

“It is reflected in the small, routine interactions like how often clinicians override 

recommendations, how those overrides are reviewed, and whether the system adapts in response. 

“It starts with clinicians being involved from the beginning, in procurement, configuration and risk design. That’s what changes the dynamic from ‘here’s a tool, use it’ to ‘here’s a tool we built for our clinical reality’.” 

Dr Luckraj said in day-to-day workflows, good governance meant clinicians had a clear pathway when the AI’s output didn’t match their clinical judgment and “they felt both empowered to challenge it and supported when they did.”

“That psychological safety is non-negotiable. If questioning the AI is seen as obstructive or critical, governance has failed regardless of what the policy says.”

She added that in practice, it also meant that overrides are treated as intelligence – not friction. 

“Feedback loops, often invisible, are what good governance actually looks like.”

“At scale, governance is not theoretical but operational, continuous and tested under real pressure. This perspective comes from having implemented AI-enabled triage at national scale and now advising organisations on how to do it safely,” Dr Luckraj said.

CANDOUR + EXCHANGE

Looking ahead to HealthTechX Asia, Dr Luckraj told Pulse+IT she was “hoping for more candour.”

“I’d love to see more people talking about what didn’t hold up in real-world use and what had to be reworked under pressure. Across different jurisdictions I’m working in, there is strong alignment on the principles of safe AI, but much less clarity on how to operationalise them at scale.”

She was also hoping for more cross-regional exchange.

“The AI governance challenges facing health systems across Southeast Asia, South Asia and the Pacific may not be identical to what I navigate in Australia, but there’s real common ground around equity of access, workforce readiness, regulatory maturity and the tension between moving fast and moving safely. We should be drawing on each other’s experience more deliberately. 

“I think we need to stop asking ‘how do we adopt AI safely?’ and start asking ‘how do we build systems that remain governable as they scale and evolve?’, where governance isn’t layered on afterwards, but designed in from the start,” Dr Luckraj said, adding that “It requires clinical leadership to be embedded much earlier and much more decisively in how these technologies are selected, configured, and deployed.”

“It’s a question I’m bringing directly into my advisory work at Nirvai and one where the sector now has a real opportunity to build greater operational clarity and maturity.”

Pulse+IT is an official media partner of HealthTechX Asia, taking place 6 – 7 May in Singapore.