Are We Testing Competence or the Quality of the Answer?

An examination can produce a correct answer without establishing that the person being examined understands it.

Two medical trainees submit identical answers to a clinical examination question.

Both responses are comprehensive, well structured and clinically appropriate. They identify the relevant diagnoses, justify the investigations and outline safe management.

One trainee constructed the answer independently. The other used artificial intelligence.

The examiner awards both the same mark.

But have they demonstrated the same competence?

The question exposes a problem medical education can no longer avoid. Examinations have traditionally relied on the relationship between the quality of an answer and the abilities of the person producing it. That relationship has never been perfect, but it has been sufficiently meaningful to support decisions about progression, certification and professional readiness.

AI makes the relationship less certain.

A trainee may now produce an excellent response without possessing all the knowledge or reasoning required to generate it independently. The answer may be entirely correct. The inference we draw about the trainee may not be.

This is fundamentally a question of assessment validity.

An assessment is useful not simply because it produces reliable scores, but because those scores support defensible conclusions about the abilities we intend to measure.

If we are examining foundational physiology, independent diagnostic reasoning or the ability to recognise a dangerous clinical presentation, unrestricted AI assistance may obscure the very competencies we are attempting to establish.

But there is another side to this argument.

Doctors do not practise in isolation from external knowledge. We consult colleagues, use calculators, follow guidelines and increasingly interact with clinical decision-support systems. The ability to use these resources appropriately is itself an important professional skill.

An examination that prohibits every form of assistance may establish independent knowledge while telling us relatively little about how a candidate performs in contemporary practice.

Neither approach is necessarily wrong.

They measure different things.

Consider an assessment in which a trainee is presented with a patient who has an apparent pulmonary embolism. An AI system recommends anticoagulation and offers a convincing explanation. However, the clinical information contains a significant contraindication that the system has overlooked.

The candidate's task is not merely to generate a management plan. It is to evaluate the recommendation, identify the unsafe assumption and justify an alternative approach.

A candidate who recognises the problem has demonstrated something important: the ability to exercise judgement when technological assistance is plausible but unreliable.

Such an assessment could be valuable. But it would not necessarily establish the candidate's independent capacity to diagnose pulmonary embolism or formulate treatment without assistance.

That distinction should shape the future of medical examinations.

Some assessments should deliberately require unaided performance. Others could permit AI while examining how candidates interrogate its outputs, reconcile conflicting evidence and recognise their own limitations. Simulated clinical encounters and oral examinations may help reveal the reasoning behind decisions, although these approaches also require careful design and validation.

The objective should not be to make examinations harder for the sake of preserving their traditional format.

Nor should it be to embrace AI simply because clinical practice increasingly uses it.

The purpose is to obtain credible evidence of competence.

This matters beyond the examination room. When a medical college certifies a specialist, that qualification communicates an assurance to patients, employers and the profession. It signifies that the individual has demonstrated abilities considered necessary for safe, independent practice.

We must remain clear about what those abilities are and how they have been established.

AI may force us to reconsider which competencies require independent demonstration and which are better assessed through effective use of technological assistance. That reconsideration is overdue in some areas and uncomfortable in others.

But it cannot be avoided simply by banning AI or allowing it everywhere.

The challenge is not that artificial intelligence can answer our examination questions. It is that we must become more precise about what those questions were supposed to prove.