FDA Draft Guidance on PCCP: Re-Training and Performance Evaluation

Table of Contents

The Food and Drug Administration (FDA or the Agency), the US regulating authority in the sphere of medical devices, has published a guidance document dedicated to  the requirements for a predetermined change control plan (PCCP) to be included in marketing submissions for medical devices utilising artificial intelligence/machine learning (AI/ML) technologies. The approach described in the document is intended to reduce the regulatory burden for the medical device manufacturers with respect to modifications to the products already placed on the market – the said approach provides that for the changes covered by a PCCP authorized by the FDA additional marketing submission will not be needed. 

It is important to mention that provisions of the present guidance are non-binding in their legal nature, nor are intended to introduce new rules or impose new obligations, but rather to provide additional clarifications regarding the existing regulatory requirements, as well as recommendations to be followed by medical device manufacturers and other parties involved in order to ensure compliance thereto. Moreover, the authority explicitly states that an alternative approach could be applied, provided such an approach is in line with the current regulatory framework and has been agreed with the authority in advance. 

The scope of the guidance covers, inter alia, the aspects related to re-training ML-based medical devices. In particular, the document addresses specific matters related to the nature of such devices to be taken into consideration in order to ensure their continued safety and proper performance.

Re-Training Objectives, Focus and Implementation

According to the guidance, the aspects to be addressed in a PCCP with respect to re-training objectives and focus include, inter alia, the following ones:

• How are the modifications presented in the Description of Modifications in the PCCP related to the planned re-training methods?
• For each part of the ML-DSF that will be modified, is ML model re-training needed to achieve the modifications specified in the PCCP?
• If re-training applies to only certain parts of the ML-DSF, what are the plans to ensure that other functions or software components are not affected? 

The document also emphasizes the importance of paying attention to specific parts of the ML-device software function (ML-DSF) subject to changes, as well as the details of modifications expected to take place. In this respect, a rationale is required for changes introduced to each of the parts. 

When it takes to the actual implementation of re-training, the aspects to be considered include specific triggers for re-training (for example, new data becoming available, or expiration of a pre-defined period), particular strategies to be followed, as well as the risks associated with re-training.

Performance Evaluation

Another important aspect covered by the scope of the present draft guidance relates to the performance evaluation of AI-powered medical devices. 

First of all, the authority emphasizes the importance of determining specific triggers that will initiate performance evaluation of the product upon implementation of modifications or re-training, as well as the frequency of evaluation to be conducted in order to ensure continuous safety and proper performance. 

In terms of metrics and elements of the assessment, the FDA encourages medical device manufacturers to pay attention to specific metrics that will be used in the context of performance evaluation. The questions to be answered here are the following:

• What metrics will be computed to understand device performance?
• How do these metrics demonstrate that the modified device can be safely used?
• How will the metrics provide a comprehensive assessment of device performance and patient safety? 
• What corner cases (i.e., cases outside the norm) or known challenging scenarios will be evaluated? 

When it takes to the performance evaluation of ML-based medical devices, the authority also mentions the matters related to a statistical analysis plan. The scope of such a plan should cover the aspects associated with:

• High-risk subpopulations and subgroups that might be affected by the changes;
• Labeling specifications;
• Ensuring the performance in one area will not affect the performance in another area;
• The way the sample size should be determined;
• The scope of population covered by the primary analysis (e.g., any subjects within the intended use population, or only eligible ones);
• The approach to be applied with respect to variability;
• The way missing data is to be addressed in the course of the analysis.

The document also outlines the applicable questions regarding performance targets which should be:

• What are the acceptance criteria?
• What clinical considerations were used to develop the acceptance criteria?
• How will the acceptance criteria support that the modification will be successfully implemented? 

In certain cases, additional testing could be required. When assessing the need for additional testing, medical device manufacturers should determine whether the database testing is sufficient to address properly the risks associated with the changes suggested. Apart from that, attention should be paid to clinical usability. 

In summary, the present draft guidance issued by the FDA describes in detail the approach to be applied with respect to re-training. The document also provides additional clarifications regarding performance evaluation and highlights the key points associated thereto.

Sources:

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/marketing-submission-recommendations-predetermined-change-control-plan-artificial

How Can RegDesk Help?

RegDesk is a holistic Regulatory Information Management System that provides medical device and pharma companies with regulatory intelligence for over 120 markets worldwide. It can help you prepare and publish global applications, manage standards, run change assessments, and obtain real-time alerts on regulatory changes through a centralized platform. Our clients also have access to our network of over 4000 compliance experts worldwide to obtain verification on critical questions. Global expansion has never been this simple.