Artificial intelligence (AI) in healthcare can be incredibly powerful…until it isn’t.
KFF Health News has published a reminder that while AI can unlock significant clinical, financial and operational efficiencies and even breakthroughs, the technology is still in its relative infancy in terms of implementation in live environments. As such, human verification and backup remains essential in many critical areas.
KFF tells the story of anesthesia staff at Erlanger Baroness in Chattanooga, Tenn., noticing a sleepy nurse slurring his words while on duty in the hospital’s surgery center. After the nurse was fired after failing a drug test, he admitted that he had been diverting and abusing leftover fentanyl after surgeries for months, frequently on a daily basis.
While that certainly isn’t an unusual tale, the difference in this case is that the hospital had been using AI-powered medication-monitoring software designed to detect missing drugs. The software, however, “failed to raise alarms, overlooking missing drugs and other inconsistencies that should have been flagged,” KFF reports.
“The Erlanger case, which has not been previously reported, offers a rare glimpse at an apparent failure of AI drug diversion software used in hundreds of U.S. hospitals with little transparency or oversight,” the media outlet writes. “Healthcare facilities are not required to disclose their implementation of this kind of software or report malfunctions to anyone, so there is no full account of how widely these programs are used or how often they fail.”
KFF Health News proceeds to take a deep dive into this issue. Read its full report here.
