When AI Gets It Wrong: Who Is Really in Control?

We are told that clinicians must remain accountable for decisions involving artificial intelligence. But accountability becomes complicated when the decisions shaping those systems are made elsewhere.

A hospital introduces an artificial intelligence system to help identify patients at risk of clinical deterioration.

The technology has been purchased, evaluated and integrated into the electronic medical record. Clinicians receive training. The system produces alerts, identifies patterns and recommends further assessment.

Its purpose is sensible. Earlier recognition of deterioration could prevent harm.

But consider what happens when the system gets something wrong.

A patient deteriorates without triggering an alert. Or an alert directs attention towards one diagnosis while an important alternative is overlooked. The treating clinician has seen the recommendation, considered it and made a decision.

An investigation follows.

Who is responsible?

We tend to answer this question by returning to a familiar principle: the clinician remains accountable for the care they provide.

That principle matters. Professional responsibility cannot simply disappear because software contributed to a decision. Patients deserve clinicians who exercise independent judgement rather than automatically accepting recommendations.

But the explanation is incomplete.

The individual doctor did not necessarily select the software, define its intended use, establish its performance thresholds or determine how prominently its recommendations would appear in the clinical workflow.

They may have had limited access to the evidence supporting its adoption. They may not know whether the system performs differently across patient populations, how frequently it produces misleading results or whether its accuracy has changed since implementation.

Yet the expectation remains that they will recognise when it is wrong.

This is the contradiction at the centre of human oversight.

Responsibility is often assigned to the person making the final decision, while substantial control over the conditions influencing that decision lies elsewhere.

Medicine has encountered similar problems before. Medication safety, infection prevention and the investigation of clinical incidents have all required us to look beyond individual mistakes towards the systems in which those mistakes occur.

Artificial intelligence deserves the same scrutiny.

Professional accountability, organisational responsibility and legal liability are related, but they are not interchangeable. A clinician may have an obligation to question a recommendation. A health service may have a separate obligation to select, validate, monitor and safely implement the system. Legal responsibility will depend on the circumstances and applicable law.

Reducing these distinctions to the phrase human in the loop risks creating the appearance of a safeguard without establishing whether it works.

The World Health Organization has warned about automation bias in healthcare AI: the tendency to place unwarranted reliance on automated recommendations, potentially overlooking errors or contrary evidence.

That risk is not resolved merely by requiring a clinician to approve the output.

Meaningful oversight requires the ability to understand a system's limitations, challenge its conclusions and act independently when necessary. It also requires an environment in which questioning an automated recommendation is practical rather than merely permitted.

A hospital cannot reasonably expect clinicians to compensate for every foreseeable weakness in a system it has chosen to deploy.

Nor should developers be absolved of responsibility simply because their technology is described as decision support. Design choices influence how recommendations are interpreted, how uncertainty is communicated and how easily users can identify conflicting evidence.

None of this diminishes the clinician's duty to the patient. It recognises that safe care depends on more than the decisions of one person.

AI may improve diagnosis, reduce errors and help clinicians manage increasingly complex information. These possibilities deserve serious consideration. But the benefits must be accompanied by an equally serious approach to evaluation, governance and the investigation of harm.

The test of responsible implementation is not whether an organisation can identify a person who signed off on the final decision.

It is whether everyone with influence over that decision has exercised the responsibilities appropriate to their role.

Accountability without meaningful control is not a safeguard. It is simply somewhere to place the blame.