As we know, a basic rule of thumb for use of artificial intelligence (AI), at least in this phase of its evolution, is that you should always have a human in the loop. AI makes mistakes, makes things up, tries to please you, and sometimes hallucinate. My Alexa at home keeps calling me the wrong name.
So, the thinking is, you’ve got to keep a human in the loop.
From a government policy perspective, especially if you’re using AI to make clinical diagnoses or decisions, it seems like a no-brainer: as “brilliant” as the AI is, let’s always make sure that there’s a clinician reviewing the AI analyses or decisions, giving a second opinion.
In fact, we should probably require it by law; that’s how the thinking goes.
As an example, there’s been backlash against a recent state-approved pilot program in Utah that allows AI to renew prescriptions for commonly prescribed medications…without a doctor in the loop.
Some physician groups, including the Utah Medical Licensing Board, think this is not a good idea at all, with a slippery slope and all that, and many people beyond physicians agree.
That’s why the discussion surrounding a recent interview in Becker’s Health IT feels so counterintuitive. The interview was with Dr. Paul Testa, chief health informatics officer with NYU Langone Health.
Dr. Testa suggested that having a doctor in the loop, in some instances, can make AI’s performance worse. That is, a clinician’s review of an AI’s decision may not be in the best interest of the patient.
Dr. Testa imagined a time when, in the not-too-distant future, an AI’s performance may outperform the clinician in some contexts. Already, numerous studies demonstrate that AI can catch issues that human clinicians miss in the areas of radiology, dermatology, and diagnostics.
What happens when we realize that safer patient care sometimes means turning the decision over to AI completely? Humans are, after all, human, and they may be swayed by their own lack of knowledge and unconscious biases when asked to review an AI’s decision.
In other words, there may come a time when the responsible thing for the patient’s health will be to NOT have a doctor in the loop.
Another element of doc-in-a-loop policies is the question of accountability. If our policy is that a doctor should be in the loop, what happens when a doctor decides the AI is wrong, but then the AI turns out to be correct? Or a doctor defers to an AI tool that turns out to be wrong? Under a doc-in-the-loop policy, the clinician would be to blame.
Take, for instance, the Food and Drug Administration’s (FDA’s) guidance on AI in clinical decisions released in January. The FDA said that there will be no oversight of AI in clinical decisions if a clinician can review the AI’s recommendation and doesn’t “rely primarily on the AI’s decision.” In a world where many if not most clinicians are already using AI in some capacity for some of their decisions, this means that the FDA holds the clinician accountable.
A number of states have passed or are considering legislation that also puts accountability on the shoulders of the clinician, explicitly or implicitly. Delaware passed a law in April that prohibits AI and other “nonhuman entities” from being licensed as a physician, doctor, or nurse. In other words, unlike the Utah pilot, Delaware does not want AI making decisions that heretofore have been the purview of licensed clinicians.
On its face, this sounds prudent and logical. It’s hard to hold a non-human to account. We need a human to blame.
That brings us to a recent think piece in the British Medical Journal (BMJ) written by doctors associated with the Mayo Clinic. The problem, at least according to the article’s authors, is that while we expect the clinician to know something about healthcare, it is a leap to expect the clinician to “reliably detect and correct errors from a complex AI system.” We are now, the authors said, expecting clinicians to “oversee and override black box algorithms and interpret outputs for each patient’s clinical, cultural, and moral reality.”
The BMJ article suggests that we (and government policy) are looking at the wrong human – or, more specifically, wrongly putting all of the accountability on a single group of humans, the doctors.
The authors suggest that accountability should also include the humans that design, procure, and maintain these tools. And that’s not just the AI developers. It should be a shared risk, the authors say, including the implementing organizations: that is, the hospitals and governance committees that oversee the use of the AI technology.
And yes, for certain tools and under certain structures, the clinician should be in the loop and share accountability.
All of this was to say that the question of who is accountable for AI analyses and decisions needs a subtle, complex response. And legislation and government policies need to reflect those subtleties.
Unfortunately, subtlety is one thing legislation is often not.
EDITOR’S NOTE:
The opinions expressed in this article are solely those of the author and do not necessarily represent the views or opinions of MedLearn Media. We provide a platform for diverse perspectives, but the content and opinions expressed herein are the author’s own. MedLearn Media does not endorse or guarantee the accuracy of the information presented. Readers are encouraged to critically evaluate the content and conduct their own research. Any actions taken based on this article are at the reader’s own discretion.
