Medical professionals are currently utilizing unregulated artificial intelligence resources, such as virtual assistants for note-taking and predictive software for disease diagnosis and treatment.

The government has been slow to regulate the rapidly advancing technology due to the significant funding and staffing obstacles that agencies like the Food and Drug Administration face when creating and enforcing rules. It is doubtful that they will catch up in the near future. This means that the implementation of AI in healthcare is becoming a risky experiment to determine if the private sector can safely revolutionize medicine without government oversight.

John Ayers, an associate professor at the University of California San Diego, expressed that the cart is currently far ahead of the horse. This presents a challenge of how to bring it back under control without risking falling into the ravine.

The FDA is looking to monitor the changes in AI software over time, which is a departure from their usual one-time approval process for medical devices and drugs. This proactive approach is new for the FDA.

In October, President Joe Biden made a commitment to a swift and organized approach from government agencies in addressing AI safety and effectiveness. However, regulatory bodies such as the FDA lack the necessary resources to oversee a constantly evolving technology.

“At a recent conference in January, FDA Commissioner Robert Califf stated that we would require a doubling of our size. However, it seems that taxpayers are not currently inclined to support such an expansion, as reiterated in a recent meeting with FDA stakeholders.”

Califf was honest about the difficulties faced by the FDA. Assessing AI is a huge undertaking due to its continuous learning and potential variations in performance based on the environment. This does not align with the current processes of the agency when approving drugs and medical devices, as they do not need to monitor their evolution.

The FDA is facing more than just the need to change its regulations or increase its workforce. According to a recent report from the Government Accountability Office, the agency desires greater authority to obtain AI performance data and establish guidelines for algorithms in a more precise manner than its current risk assessment system allows for drugs and medical devices.

Given that Congress has just started to discuss and has not yet come to an agreement on regulating AI, the process could be lengthy.

Historically, Congress has been reluctant to grant the FDA more power. Currently, the FDA has not made any requests for expanded authority.

The agency has provided instructions to manufacturers of medical equipment on how to properly integrate artificial intelligence, causing opposition from technology companies who believe the agency has exceeded its authority – despite the fact that the instructions are not legally enforceable.

Meanwhile, there are AI professionals in both academia and industry who argue that the FDA is not utilizing its existing powers adequately.

Scope of authority

The progress of AI has resulted in significant discrepancies in the scope of regulation by the FDA. It does not evaluate tools such as chatbots and lacks jurisdiction over systems that condense doctors’ notes and carry out essential administrative duties.

The FDA oversees first-generation AI tools in the same way it oversees medical devices. 14 months ago, Congress gave the agency the authority to approve planned updates for device makers, including those utilizing early AI, without requiring them to reapply for clearance.

The extent of the FDA’s authority concerning AI is not yet determined.

A group of companies submitted a formal request to the FDA, claiming that the agency overstepped its boundaries by issuing a 2022 guideline stating that manufacturers of artificial intelligence providing time-sensitive recommendations and diagnoses must obtain FDA approval. Despite the guideline not being legally binding, companies often feel obligated to adhere to it.

also expressed confusion over the scope of FDA authority and how power over AI regulation is split among it and other agencies within the Department of Health and Human Services, like the Office of the National Coordinator for Health Information Technology. That office set rules requiring more transparency around AI systems in December.

According to Colin Rom, a former senior adviser to FDA Commissioner Stephen Hahn and current leader of health policy at venture capital firm Andreessen Horowitz, the lack of guidance from HHS creates confusion for individuals in the industry who are unsure of where to seek clarification.

The FDA informed GAO that in order to actively monitor the long-term effectiveness of algorithms, it requires additional authorization from Congress to gather performance data.

The agency expressed a desire for enhanced authority to establish tailored measures for individual algorithms, rather than relying on current medical device classifications to determine regulations.

The FDA intends to convey its requirements to Congress.

Oversight outsourced

However, this still leaves it dependent on a Congress that is unable to reach a consensus.

Therefore, Califf and other industry professionals have suggested an alternative solution: the establishment of public-private assurance labs, potentially at prominent universities or academic health facilities, to evaluate and oversee the use of artificial intelligence in healthcare.

During last month’s consumer electronics show, Califf stated that there needs to be a group of entities responsible for conducting assessments to ensure that algorithms are certified and not causing harm.

There is also backing for the concept among members of Congress. Senator [last name] has expressed support for it as well.John Hickenlooper

A representative from Colorado is urging for skilled outside sources to conduct evaluations on sophisticated artificial intelligence. The focus is on generative AI, such as ChatGPT, which imitates human intelligence. This aligns with the oversight framework proposed by Califf.

This method may have shortcomings, as mentioned by several AI professionals, as AI that is tested on a large university campus may not be as effective in a small rural hospital setting.

During a recent Senate Finance Committee hearing on artificial intelligence in health care, Mark Sendak, the lead for population health and data science at Duke University’s Institute for Health Innovation, emphasized the importance of adapting to varying environments as a practicing physician. He highlighted the need for each healthcare organization to have the ability to independently regulate the use of AI.

In the Journal of the American Medical Association, Micky Tripathi, the national coordinator for health information technology, and Troy Tazbaz, FDA’s director of digital health, stated that assurance labs must consider this issue in January.

A group of researchers from Stanford Medicine, Johns Hopkins University, and the Mayo Clinic collaborated on an article proposing the establishment of a few pilot labs to develop validation systems.

However, smaller players remain concerned about potential conflicts of interest if the pilot labs are affiliated with organizations that also produce their own AI systems or work with technology companies, despite efforts by regulators, major universities, and health care providers to collaborate and instill confidence.

Ayers believes that the FDA should take charge of validating AI within its own organization. He also suggests that creators of AI systems should be required to demonstrate their ability to enhance patient outcomes, regardless of who is responsible for oversight.

The author observed that an AI program created by Epic, a company that manages electronic health records, did not recognize sepsis, a potentially deadly response to infection, which went unnoticed by regulators. The company has since made changes to its algorithm, and a representative from the FDA stated that they do not reveal their correspondence with individual companies.

However, the event has caused numerous individuals in the healthcare and technology fields to believe that the agency is not effectively utilizing its current powers.

Ayers stated that they should be actively monitoring and regulating this situation.