By AI Trends Staff
August 12, 2021 | The US Food and Drug Administration (FDA) in February issued an AI and Machine Learning Software as a Medical Device Action Plan, outlining the way forward for manufacturers seeking the agency’s approval for their innovative medical devices incorporating AI.
“Artificial intelligence (AI) and machine learning (ML) technologies have the potential to transform health care by deriving new and important insights from the vast amount of data generated during the delivery of health care every day,” state the report authors at the outset.
The report is a follow-up to a 2019 effort to begin the discussion of how to proceed with medical devices using AI, in response to requests that the agency update its approval process. “The FDA’s traditional paradigm of medical device regulation was not designed for adaptive artificial intelligence and machine learning technologies,” state the FDA plan authors.
The FDA action plan looks for a commitment from manufacturers as to how the device will perform, with a process for periodic updates as the device is developed. In this way, the FDA and manufacturers would be able to evaluate the software product from premarket development to post-market performance. This would enable the FDA to embrace the power of AI and ML-based software in a medical device, while ensuring patient safety.
“This action plan outlines the FDA’s next steps towards furthering oversight for AI/ML-based SaMD [Software as a Medical Device],” stated Bakul Patel, director of the Digital Health Center of Excellence for the FDA, stated when the action plan was released, according to an account in HealthITAnalytics.
“The plan outlines a holistic approach based on total product lifecycle oversight to further the enormous potential that these technologies have to improve patient care… To stay current and address patient safety and improve access to these promising technologies, we anticipate that this action plan will continue to evolve over time,” he stated.
Review Finds FDA Medical Device Testing to be Limited So Far
Researchers at Nature decided to evaluate how well the FDS is addressing issues of test data quality, transparency, bias, and algorithm monitoring in practice, according to the HealthITAnalytics account. The Nature team aggregated 130 AI devices approved by the FDA between January 2015 and December 2020, and assessed how well they performed.
The review showed that 126 of the 130 AI devices underwent only retrospective studies at their submission, that is, information collected about the past. None of the 54 high-risk devices were evaluated by prospective studies that would look at data over time.
“More prospective studies are needed for full characterization of the impact of the AI decision tool on clinical practice, which is important, because human–computer interaction can deviate substantially from a model’s intended use,” stated the Nature report authors.
Of the 130 AI devices analyzed, 93 devices did not have publicly reported multi-site assessment as part of the evaluation study, the Nature researchers found. Of the 41 devices with the number of evaluation sites reported, four devices were evaluated in only one site and eight devices were evaluated in only two sites.
“This suggests that a substantial proportion of approved devices might have been evaluated only at a small number of sites, which often tend to have limited geographic diversity,” the researchers noted.
Necessary Data to Build AI SaMD Challenging to Assemble
A recent report from the Pew Charitable Trust outlined some challenges facing AI SaMD manufacturers seeking FDA approval.
For example, AI algorithms need to be trained on large, diverse datasets to work effectively across a variety of populations and settings, and to ensure they are not biased. “However, such datasets are often difficult and expensive to assemble because of the fragmented U.S. healthcare system, characterized by multiple payers and unconnected health record systems,” state the Pew report authors.
An analysis conducted in 2020 of data used to train image-based diagnostics AI systems found that some 70% of the included studies used data from three US states; 34 states were not represented at all. This would risk missing variables such as disease prevalence and socioeconomic differences, the report authors noted.
“Moreover, assembling sufficiently large patient datasets for AI-enabled programs can raise complex questions about data privacy and the ownership of personal health data,” the Pew authors stated. They noted some startups developing their AI-based programs are using patient data sometimes without the consent of the patient, ensnaring them in an ongoing debate around consent, and whether patients who do share data should share in profits.
In their conclusion, the Pew authors stated, “The FDA is attempting to meet these challenges and develop policies that can enable innovation while protecting public health, but there are many questions that the agency will need to address in order to ensure that this happens.”
Medical Device Manufacturer Cites Promising Uses for SaMD
An experienced medical device manufacturer knows the potential of a breakthrough technology. Whoever for example is able to develop an effective early detection tool for lung cancer would be in a good position. “Using AI as an early detection tool has a strong probability of being a game changer,” stated Lorenzo Gutierrez, Microfluidic Manager for StarFish Medical, a medical device design, development and contract manufacturing company based in British Columbia, Canada, in a blog post.
That outcome seems very possible. According to a 2019 study cited by Gutierrez, a deep learning algorithm achieved a lung cancer detection performance of 94.4% based on 6,716 cases. That result outperformed human radiologists by an 11% reduction in false positives and a five percent reduction in false negatives.
In the medical device space, he identified promising target uses for AI SaMD, including:
Diagnosis of heart diseases. A machine learning algorithm (myocardial-ischemic-injury-index) incorporating age and sex paired with high-sensitivity cardiac troponin I concentrations was used to train an AI platform utilizing data from 3,013 patients. The platform was then tested on 7,998 patients with suspected myocardial infarction. It was found to outperform physicians, with a sensitivity of 82.5% and a specificity of 92.2%.
Detecting retinopathy. Diabetic retinopathy (DR) is one of the leading causes of preventable blindness globally. In a study published by American Academy of Ophthalmology, a total of 75,137 publicly available fundus images from diabetic patients were used to train and test an artificial intelligence engine to differentiate healthy fundi from those with DR. The results showed an impressive 94% and 98% sensitivity and specificity, respectively.
Biosensors for monitoring vital signs. Biosensor-based devices generate huge data sets. AI could be used to predict the trends and the probability of disease occurrence. The integration of AI in cardiac monitoring-based biosensors for point of care (POC) diagnostics are a good example. Machine-learning algorithms are used with microchip-based cardiac biosensors for real-time health monitoring and to provide accurate clinical decisions in a timely manner.
This summary also ran in AI Trends. Read the source articles and information in the AI and Machine Learning Software as a Medical Device Action Plan from the FDA, in HealthITAnalytics, from the Pew Charitable Trust and in a blog post from StarFish Medical.