An AI agent has found its way into Australian health data. It is a moment for health leaders to lead – and proof that the runway to December is the right runway, writes Luli Adeyemo.

In June, the same month Australia’s updated AI policy reached its first milestone, an AI agent went looking for answers about Australia and ended up somewhere it was never meant to be.

Last week we learnt that an OpenAI agent, during the company’s own testing, got inside a government portal holding Medicare statistics. Nobody instructed it to. And months passed before the people responsible for that data heard about it.

It happened here. And it happened in health.

I want to resist two easy reactions. The first is panic. From what we know so far, this was a statistics portal, not patient records, and the investigation is in capable hands. The second is shrugging it off as a one-off. It is not a one-off. It is a preview.

The detail that matters most is the simplest one. The agent did something its makers did not intend. Every health leader should sit with that for a moment.

Intent is a human job

Earlier this year I wrote for Pulse+IT that the algorithm does not face the coroner. The same principle applies here. An agent cannot be a named accountable owner. It cannot sit before an inquiry or explain itself to a patient. People do that. Intent is a human quality, and so is accountability.

That is what makes agents different from the AI tools most of us have been governing until now. A tool gives an answer, and a person decides what to do with it. An agent decides, and acts. The governance question moves from “is this output right?” to “what is this thing allowed to touch, and who is watching when it does?”

It is also worth noticing how ordinary the starting point was. This was not an attack. It was a capable system, set a routine research task, pursuing it further than anyone expected. That is precisely why it matters. The risk did not come from malice. It came from a goal, with nobody close enough to the edges to see where it was heading.

Now imagine that same eagerness pointed at a booking system, a referral queue or an elective surgery waitlist. An agent that is simply trying to be helpful can still end up deciding who goes first.

The influence gap, in a new form

One thought has stayed with me. Australian health data became part of someone else’s test. The people who look after that data had no part in designing the test, setting its boundaries, or deciding how far a search for an answer was allowed to go.

That is the influence gap, arriving in a new form. As I wrote here earlier this month, when the rooms where AI is designed, tested and governed reflect only a narrow band of perspectives, whole categories of risk stay invisible until someone else meets them. With pulse oximeters, it was patients with darker skin. This time, it was an Australian health data portal, a long way from the room where the test was designed.

Then there is the question of how we hear about it. So here is one worth asking in every health organisation this week: if a vendor discovered tomorrow that its AI had been inside your systems, where would that email land, and who would read it? A shared mailbox is not a governance structure. Disclosure is only as good as the named person it reaches, and how quickly they can act.

Capability, not just rules

Some will read this week as proof that we need more rules. I understand the instinct, and international cooperation on AI matters. But rules agreed anywhere still need custodians at home to hold them: in our hospitals, our health departments and our data agencies.

In June I argued that a decision to govern through judgement rather than statute does not lessen the need for governance. It concentrates it. This incident shows exactly where it concentrates: in named owners, clear escalation paths, and people with the capability to recognise what an agent is doing and ask the hard question early. That is not a case against Australia’s light-touch model. It is a clear description of what makes it work.

And the timing is, again, on our side. The deeper accountability requirements in Australia’s AI policy, including registers of where AI is used and named accountable owners, become compulsory in December. Here is how I would use the weeks that remain.

Extend your AI register beyond the tools you have bought. Include the agents acting on your behalf, and think hard about the agents that may reach your public-facing systems from outside.

Name the owner and the door. Decide who receives an AI incident disclosure, whether it comes from a vendor, a researcher or your own staff, and how fast it reaches someone who can act.

Widen the room where agent permissions are set. Clinicians, privacy and data custodians, consumers, First Nations and culturally diverse voices: the people most likely to ask the question nobody else thought of, which is usually “what else could this touch?”

There will be more incidents like this, and some will be more serious. 

That is why the AI Governance Practitioners Programme, designed by Dr Kobi Leins, exists: to put capable, diverse, trained custodians of AI into the rooms where these permissions are decided, before an agent tests the boundaries for us.

The agent did not mean to. That is exactly the point.

AI First, Human Always. We are all custodians of AI, and this week the custody question knocked on our front door. The runway to December just got a little shorter. Let’s use every day of it.