The article describes in detail the regulatory requirements for certain innovative medical devices.
Table of content
Health Canada, a Canadian regulating authority in the sphere of healthcare products, has published a guidance document dedicated to machine learning-enabled medical devices. The document provides an overview of the applicable regulatory requirements, as well as additional clarifications and recommendations to be taken into consideration by the parties involved in order to ensure compliance.
At the same time, provisions of the guidance are non-binding in their legal nature, nor are they intended to introduce new rules or impose new obligations. The authority also reserves the right to make changes to the guidance and recommendations provided therein, should such changes be reasonably necessary to reflect corresponding amendments to the underlying legislation.
The document outlines Health Canada’s current approach to the regulation of machine learning‑enabled medical devices (MLMD). It describes how manufacturers should prepare applications by providing robust evidence of safety and effectiveness across the full product lifecycle.
The guidance applies to Class II, III, and IV devices that use machine learning (ML) either wholly or in part to achieve a defined medical purpose, as per the definitions in the Food and Drugs Act and accompanying Medical Devices Regulations.
Introduction
Artificial intelligence (AI) is an overarching term for algorithms and models capable of performing tasks such as learning, decision making, and prediction. Within this broad field, machine learning (ML) specifically refers to methods where training algorithms are applied to data to establish mathematical models without relying on explicit programming.
When these techniques are incorporated into devices for medical purposes, they become known as ML‑enabled medical devices (MLMD). According to the guidance, MLMDs are regulated under the same legal framework as other medical devices, which means that they must comply with the statutory and regulatory requirements designed to ensure patient safety and device efficacy.
A key element highlighted in this guidance is transparency – that is, the degree to which clear, appropriate information is provided to all stakeholders (including patients, healthcare providers, and regulators) regarding the device’s function, risks, and performance. This transparency plays a vital role in informed decision-making and in maintaining high safety standards throughout the device lifecycle.
An innovative aspect introduced in this guidance is the concept of a predetermined change control plan (PCCP). A PCCP offers a mechanism for obtaining regulatory pre‑authorization for planned changes in an ML system, addressing known risks in a timely manner while maintaining safety and performance standards.
In circumstances where uncertainties or risks are identified, Health Canada may require additional terms and conditions on the medical device licence to safeguard ongoing safety and effectiveness.
Scope and Application
This guidance is specifically intended for manufacturers who are submitting new applications or amendments for ML‑enabled medical devices classified as Class II, III, or IV. It focuses on the ML system component of an MLMD and does not extend to non‑ML information that may be part of a complete medical device licence application.
It is important to mention that while this document addresses ML‑related aspects, the manufacturers should also consider and comply with other relevant guidance documents. These include recommendations for clinical evidence, cybersecurity, and interpretations of significant changes in medical devices.
In essence, this guidance provides a framework for ensuring that the ML aspects of the device are robust, safe, and in line with current regulatory expectations.
Policy Objective
The primary objective of this guidance is to outline the supporting evidence and information that manufacturers should provide when demonstrating that their MLMD meets safety and effectiveness requirements. This encompasses both the initial application for a new licence and any subsequent amendments.
By applying a risk‑based approach throughout the product lifecycle, manufacturers are expected to ensure that:
- The MLMD continuously meets the applicable safety and effectiveness standards.
- There is a high level of protection for patient health and safety.
- Risks are appropriately managed and mitigated relative to the benefits provided to the patient.
Ultimately, the guidance seeks to foster an environment in which innovative MLMD technologies can be safely integrated into clinical practice, while also maintaining a rigorous regulatory oversight that adapts to the evolving nature of ML technology.
Policy Statements
Health Canada’s policy on ML‑enabled medical devices can be summarized by the following key points:
- Device Nature and Classification: An MLMD may exist as standalone software that qualifies as a medical device or as a component of a larger medical device. The risk classification for such devices can range from Class I to Class IV, with specific regulatory requirements applied accordingly.
- Disclosure Requirements: Manufacturers must clearly state in their cover letters that the device uses ML. If the device includes a PCCP, this must also be clearly identified. Failure to include these statements can result in delays during the review process.
- Justification of Classification: Each application must include a thorough justification for the chosen medical device classification. This involves referencing the classification rules set forth in the relevant schedules of the regulations.
- Objective Evidence: Applications must include objective evidence that demonstrates the MLMD’s intended use, along with comprehensive data supporting the device’s safety and effectiveness. This evidence should be derived from scientifically valid methodologies and must be representative of the intended patient population.
These policy statements ensure that manufacturers provide clear, transparent, and justified documentation, thereby enabling a robust review process that focuses on patient safety and device performance.
Guidance for Implementation
Health Canada’s guidance for implementing these regulatory expectations is comprehensive.
It considers the entire lifecycle of an MLMD – from initial design through post‑market monitoring – and highlights several key components:
Good Machine Learning Practice (GMLP)
Good machine learning practice (GMLP) is the foundation for ensuring that MLMD are designed, developed, evaluated, and maintained according to high-quality standards. Manufacturers are encouraged to embed GMLP principles throughout their operations. This involves:
- Integrating quality practices within the organizational framework.
- Documenting how GMLP is applied at every stage of the MLMD lifecycle.
- Demonstrating that any predetermined change control plans (PCCPs) will be executed in accordance with established protocols.
By complying with GMLP, manufacturers can mitigate risks and ensure that ML systems operate safely and effectively over time.
Design
The design phase is critical and requires manufacturers to provide detailed information on several key aspects:
Indications for Use and Intended Purpose
Manufacturers must clearly articulate the intended use and indications for use of the MLMD.
This includes specifying:
- The medical purpose (such as diagnosis, treatment, or monitoring).
- The target patient population.
- The environment in which the device is intended to operate (e.g., clinical settings, home care).
Device Description and Functionality
A comprehensive description of the device must be provided, detailing:
- The fact that the device utilizes machine learning, as explicitly stated in the cover letter.
- The ML methods employed (such as supervised, unsupervised, semi‑supervised, or reinforcement learning).
- The specific ML training algorithms (for example, convolutional neural networks, logistic regression, support vector machines, generative adversarial networks, transformers).
- The architecture of the ML system, including components, operating parameters, and tuning approaches.
- An explanation of how the ML system processes input data to generate outputs and how these outputs are integrated into the healthcare workflow.
- Information about required device inputs, hardware and software specifications, and any compatible devices.
Predetermined Change Control Plan (PCCP)
As explained by Health Canada, a pivotal innovation in this guidance is the introduction of the PCCP. This plan allows for the pre‑authorization of planned changes that address known risks – changes that would otherwise necessitate a full medical device licence amendment. The PCCP comprises three key components:
- Change Description: Documentation that outlines the baseline device performance, lists the specific changes proposed, and provides details such as the rationale, triggers, and anticipated frequency of changes.
- Change Protocol: A set of policies and procedures that govern how the changes will be implemented. This includes plans for data management, ongoing risk management, verification and validation methods, and update procedures.
- Impact Assessment: An evaluation of the potential benefits and risks associated with the planned changes, ensuring that the MLMD will continue to operate safely within its intended use.
This structured approach enables manufacturers to manage device evolution proactively while maintaining regulatory compliance and patient safety.
Risk Management
Effective risk management is integral to the lifecycle of an MLMD. Manufacturers must carry out risk assessments that cover:
- Identification of Risks: These include erroneous outputs (such as false positives or negatives), biases in output, overfitting, underfitting, automation bias, and issues like alarm fatigue.
- Risk Control Measures: Manufacturers must detail the controls in place to mitigate these risks. This includes techniques for continual monitoring, data management procedures, and strategies for managing the impact of potential performance degradation.
- Risk Analysis Methodologies: The guidance recommends established methods (such as those outlined in ISO 14971) to categorize and evaluate risks, ensuring that all potential hazards are adequately addressed.
- Integration with PCCP: For devices that utilize a PCCP, the risk management process must consider how pre‑authorized changes might influence the overall risk profile of the device.
Hence, by incorporating a rigorous risk management framework, manufacturers can demonstrate that the MLMD will consistently meet the required safety and performance standards throughout its lifecycle.
Data Selection and Management
Due to the nature of such products, the quality of the datasets used to develop and validate an ML system is crucial to its performance.
This section requires manufacturers to provide detailed information on:
- Dataset Characteristics: Description of training, tuning, and test datasets including sample sizes, demographic statistics, and clinical characteristics. A comparison between the dataset prevalence and that of the intended patient population is essential.
- Data Collection Methods: Information on the environments and devices used to collect data, whether data were gathered from single or multiple centres, and whether data were personalized.
- Justifications for Data Choices: Manufacturers should offer justifications for the chosen datasets, including statistical considerations and relevance to the intended use. Special attention should be given to identity factors (such as sex, gender, race, and age) and to the inclusion of under‑represented populations.
- Data Quality Assurance: A description of how data integrity and accuracy are maintained. This includes any techniques used for data augmentation or imputation, as well as methods to control bias in the dataset.
These detailed data selection and management practices help ensure that the ML system is trained on representative and high‑quality data, thereby supporting the device’s overall safety and effectiveness.
Development and Training
During the development phase, manufacturers are expected to provide comprehensive details on how the ML system is built, trained, and tuned.
Key points include:
- Methodology Description: A detailed account of the ML development processes, including training and tuning methods. The rationale behind selecting particular algorithms and techniques should be clearly stated.
- Reference Standard: An explanation of the reference standard used in training and tuning, including how it was defined, the justification for its selection, and any inherent limitations.
- Input Parameters and Feature Extraction: Detailed descriptions of the inputs used in developing the ML model, as well as any features derived from the input data.
- Alignment with Intended Use: A justification for the chosen development methods and how they support the MLMD’s intended use.
This level of detail not only aids in the regulatory review process but also enhances transparency, ensuring that stakeholders understand the underlying principles and assumptions of the ML system.
Testing and Evaluation
According to the guidance, robust testing and evaluation are essential to validate the performance of the ML system.
Manufacturers should include:
- Testing Methods: A description of the testing protocols used to evaluate the ML system’s performance. This might include bench testing, software verification and validation, or performance/bench testing integrated into the overall device evaluation.
- Test Data: Details regarding the datasets used for testing, ensuring consistency with the data selection and management section.
- Reference Standard for Testing: A characterization of the reference standard used during testing, along with a justification for its use and an explanation of any uncertainties.
- Performance Metrics: A clear presentation of the chosen performance metrics, acceptance criteria, and operating thresholds, accompanied by clinical and risk‑based justifications.
- Evidence of Performance: Data and evidence that demonstrate the ML system performs as intended when integrated into the full medical device system. This includes subgroup analyses and results from robustness testing (e.g., testing with unexpected inputs).
- Version Control: A detailed explanation of the software version numbering system to ensure that each version of the ML model is traceable and that any changes can be tracked over time.
Thus, by providing detailed testing and evaluation evidence, manufacturers can demonstrate that their MLMD meets the required performance standards in real‑world conditions.
Clinical Validation
Under the general rule, for devices classified as Class III or IV, clinical validation is a critical element of the application process.
This section requires:
- Clinical Study Descriptions: Detailed accounts of clinical validation studies, including the type of study performed, study design, and statistical methods used.
- Rationale and Relevance: An explanation of why the chosen study design is relevant to the MLMD’s intended use, including comparisons to the standard of care.
- Study Population: A characterization of study participants, ensuring that the study population is independent of the data used in ML system development. It should also confirm that the study population represents the intended patient demographic, including considerations for sex, gender, race, and age.
- Study Results and Limitations: Presentation of study outcomes, including subgroup analyses and discussions of any limitations. References to published clinical data may be used to support the evidence.
- Clinical Evidence Justification: A summary that ties the clinical data back to the intended use, demonstrating that the ML system is safe, effective, and suitable for clinical use.
Transparency
Transparency is a core principle throughout the MLMD lifecycle. It requires that all relevant information about the device is communicated clearly to stakeholders.
Key transparency measures include:
- Labelling and Documentation: Clear labelling that outlines the intended use, operating principles, and any limitations of the ML system. This should include detailed instructions for use, such as calibration, local validation, and interpretation of outputs.
- Explanation of ML Functionality: Detailed descriptions of how the ML system operates, including the algorithms, training methods, and the factors influencing its outputs. For example, manufacturers should explain how feature attributions, confidence scores, or saliency maps assist in interpreting the results.
- Communication of PCCP and Changes: Information regarding any predetermined change control plans and how changes will be managed over time. Users should be informed of update frequencies, labelling revisions, and any impacts on device performance.
- Structured Summaries: The use of model cards or data cards to provide a concise summary of key ML system information. This structured information aids stakeholders in understanding the device’s capabilities and limitations.
Post-market Monitoring
Once an MLMD is approved for market use, continuous monitoring is essential to confirm that it remains safe and effective.
Manufacturers are expected to implement comprehensive post-market surveillance plans that include:
- Performance Surveillance: Ongoing monitoring of the ML system’s performance, with particular attention to potential degradation over time or shifts in the input data distribution.
- Feedback and Incident Reporting: Mechanisms to collect user feedback, manage complaints, and report any adverse incidents. This ensures that any emerging risks or performance issues are quickly identified and addressed.
- Linkage to Risk Management and PCCP: Post-market monitoring should be integrated with the initial risk management framework and any PCCP strategies. This facilitates the timely implementation of corrective actions if the ML system’s performance deviates from expected parameters.
- Inter‑compatibility Checks: Regular assessments to verify that the ML system remains compatible with other hardware and software components, ensuring that system updates or environmental changes do not adversely affect performance.
This vigilant post-market oversight is crucial for maintaining high standards of safety and effectiveness throughout the device’s operational lifespan.
Licence Terms and Conditions
In addition to the pre‑market evidence required, manufacturers must adhere to licence terms and conditions imposed by regulatory authorities.
These terms may include:
- Ongoing Testing Requirements: The need for periodic tests to confirm that the device continues to meet safety and performance standards.
- Submission of Test Results: Obligations to submit protocols and outcomes of ongoing tests, ensuring continuous regulatory oversight.
- Amendments in Response to New Developments: The possibility that licence terms will be amended as new evidence emerges or as the device undergoes significant changes.
Such conditions help ensure that MLMD remains compliant with safety and effectiveness standards even after they enter the market.
Conclusion
In summary, this pre‑market guidance represents Health Canada’s current thinking on regulating ML‑enabled medical devices. It emphasizes a lifecycle approach that integrates good machine learning practices, rigorous risk management, detailed data management, comprehensive testing and evaluation, and continuous clinical validation. Additionally, it underscores the importance of transparency – both in communicating device functionality and in managing changes via a predetermined change control plan (PCCP).
Manufacturers are encouraged to incorporate these principles throughout their device development and submission processes. Doing so not only facilitates regulatory approval but also helps ensure that MLMDs are safe, effective, and able to deliver reliable performance in clinical settings. Moreover, the guidance recognizes the importance of incorporating a sex and gender‑based analysis plus (SGBA Plus) to account for the diverse anatomical, physiological, and social characteristics of patients. This inclusive approach is critical to achieving equitable health outcomes across Canada’s diverse population.
As ML technology continues to evolve, this guidance may be updated to reflect emerging best practices and technological advancements. Manufacturers are advised to remain engaged with regulatory processes, including pre‑submission consultations, to ensure that any innovative changes – such as those managed under a PCCP – are seamlessly integrated without compromising device safety or efficacy.
How Can RegDesk Help?
RegDesk is an AI-powered Regulatory Information Management System that provides medical device companies with regulatory intelligence for over 120 markets worldwide. It can help you prepare and publish global applications, manage standards, run change assessments, and obtain real-time alerts on regulatory changes through a centralized platform. Global expansion has never been this simple.
