The new article addresses specific factors related to empirical analysis (formative evaluation).
Table of Contents
The Food and Drug Administration (FDA or the Agency), the US regulating authority in the sphere of healthcare products, has published a guidance document dedicated to human factors and usability engineering in the context of the design and development of medical devices. The document is intended to provide additional clarifications regarding the existing regulatory framework, as well as recommendations to be considered by medical device manufacturers to ensure compliance with the respective regulatory requirements. In particular, the guidance highlights the most important aspects the manufacturers shall pay attention to when developing new products. It is important to mention that due to the non-binding nature of the guidance, it is not intended to introduce new rules or requirements to be followed. Moreover, an alternative approach could be applied, provided such an approach complies with the regulatory requirements and has been agreed with the authority in advance.
Formative Evaluation: Key Points
As described in the previous article, the present FDA guidance provides an overview of methods and approaches to be applied for preliminary analysis and evaluation including, inter alia, both analytical and empirical approaches to identifying critical tasks. The scope of the guidance also covers formative evaluations that, according to the guidance, are used to inform device user interface design while it is in development. As further explained by the FDA, this type of evaluation should be focused on the matters identified in the course of the preliminary analysis, including the ones related to the way potential customers will interact with the product when using it for its intended purpose. Apart from this, it should also address the aspects related to the design elements that are currently under development and are not finalized. According to the guidance, formative evaluation should be supplemental to analytical approaches described in previous articles and allow to identify the issues that in most cases cannot be identified in any way other than through observing user interaction with the device. Hence, the said approach could be used to identify in an efficient way the new critical tasks and safety-related issues. As explained by the FDA, formative evaluation could be also applied to:
- Inform the design of the device user interface (including possible design trade-offs),
- Assess the effectiveness of measures implemented to reduce or eliminate use-related hazards or potential use errors,
- Determine training requirements and inform the design of the labeling and training materials (which should be finalized before human factors validation testing), and
- Inform the content and structure of the human factors validation testing.
The choice of specific methods to be used should be based on the particular aspects to be evaluated in the course of the assessment. Hence, the scope of evaluation could differ depending on the matters subject to review, the information needed, the nature of the device in question, and other specific aspects. About medical devices to be used for formative evaluation, the authority mentions that manufacturers may use simple mock-up devices, preliminary prototypes, or more advanced prototypes as the design evolves. Apart from this, the scope of evaluation could cover particular accessories or components of the product, or even specific use conditions or population.
Specific Features
During this stage of the development process, the manufacturer may introduce changes based on the assessment and then conduct an evaluation once again to evaluate the impact caused by these changes. Thus, an iterative approach could be applied. The authority emphasizes the importance of this approach as the interface design issues identified at this stage could be easily corrected, and such a correction will require less time and resources than the same correction performed about the final device.
As further explained by the FDA, the actual effectiveness of formative evaluation depends significantly on its quality. The authority provides an example of the situation when personnel of the manufacturing facility is involved in testing intended to assess human factors. In such a case, the actual level of knowledge and experience would exceed the same levels for laypersons who are supposed to use the device, and this would impact significantly the accuracy and reliability of the results, even though such testing is not explicitly forbidden under the applicable regulations. Since persons involved in testing are already familiar with the device, its primary mode of action, and the way it should be used, the way they will interact with the device’s interface when using it could be different from the approach the laypersons will apply. This example illustrates one of the factors to be considered by medical device manufacturers when deciding on the way to evaluate as described herein.
According to the guidance, the protocol for a formative evaluation should cover the following aspects:
- Evaluation purpose, goals, and priorities;
- A portion of the user interface to be assessed;
- Use scenarios and tasks involved;
- Evaluation participants;
- Data collection method or methods (e.g., cognitive walk-through, observation, discussion, interview);
- Data analysis methods; and
- How the evaluation results will be used.
Upon completion of formative evaluation, the medical device manufacturer should be able to determine whether design changes are to be implemented to ensure the safety and effectiveness of the device concerning the interface and the way customers interact with it when using the device for its intended purpose. As it was mentioned before, such implementation of such changes is not usually associated with significant efforts since all the changes are implemented during the development stage which is beneficial from the processes’ efficiency standpoint in comparison to the situation when such changes are to be made to a final medical device.
Thus, formative evaluation could be applied to understand the way the customer interacts with a medical device in question when using it, and also to identify potential issues that could arise due to human factors, while the actual effectiveness of such an evaluation depends on its quality.
Formative Evaluation Types
According to the guidance, there are two main types of formative evaluation, namely:
- Cognitive walk-through.
- Simulated-use testing.
In summary, the present FDA guidance highlights the most important aspects related to the formative evaluation, and also describes the way it should be conducted to ensure the accuracy and reliability of its results. The document pays special attention to the factors that could potentially impact the quality of evaluation and provides recommendations to be considered by medical device manufacturers in this respect.
Sources:
How Can RegDesk Help?
RegDesk is a next-generation web-based software for medical device and IVD companies. Our cutting-edge platform uses machine learning to provide regulatory intelligence, application preparation, submission, and approvals management globally. Our clients also have access to our network of over 4000 compliance experts worldwide to obtain verification on critical questions. Applications that normally take 6 months to prepare can now be prepared within 6 days using RegDesk Dash(TM). Global expansion has never been this simple.
