3 Steps to an FDA-Worthy Human Factors Validation Study
A human factors validation study is an exercise to prove a market-ready product’s usability and safety in the hands of representative end users.
You may know this as summative usability testing. However, the FDA dislikes this term as it can “omit essential components of human factors validation testing” as described in their guidance document.
At any rate, human factors validation testing shows you’ve done your due diligence to mitigate identified hazards and the resulting harms your device might cause users and patients.
For the majority of medical devices, this study is not simply “nice to have.” It’s a critical component of your FDA submission. And if your human factors validation study isn’t done well (or at all) your submission could be flagged. Then, you must backtrack to make your product as safe and effective as necessary for clinical use, per the FDA’s standards.
So, how can you conduct a successful human factors validation study? For one, it will go smoother if you plan for it early in development. However, if you’re already farther along, don’t worry — you can still complete a satisfactory test.
Here are three necessary steps:
Step 1 – Preparation, Planning and Study Design
To keep your human factors validation study on the right path, start by partnering with a firm that specializes in medical device usability testing. Or engage with your internal human factors team. In either case, this isn’t an activity for one of your marketers to tackle. It requires a very specific skill set and testing procedure.
In addition to human factors specialists, be sure to assign the right core team of internal people to support preparation for the study. We recommend including at least one person from engineering, marketing, regulatory affairs and training as well as a point person to manage the effort.
Once you’ve gathered the appropriate people, your team must determine the user group(s) required to test your product. Next, create a participant screener for each one. Do this as soon as possible. That way, you can jumpstart the three to four week recruitment process.
The FDA expects you to test at least 15 people for each user group, but we recommend recruiting an additional three in case people drop out. Finding and scheduling participants is the single most difficult task in human factors validation testing, so we highly recommend using a recruiting agency. The more specialized your users are (e.g. a particular surgeon versus a generalist nurse), the harder it will be to recruit them. And the more you’ll need to pay them for their time.
Designing the Human Factors Validation Study
When preparing for the study, your team will review documentation such as:
-
Use Failure Modes and Effects Analysis (uFMEA)
-
Intended use information about users, training, use environment, etc.
-
Labeling documentation, including user manual, Instructions for Use (IFU), Quick Reference Guide (QRG) and Quick Start Guide (QSG)
By reviewing this documentation, you’ll see which hazardous situations could result in a harm to the user. Those are the safety-critical tasks — the use scenarios — you’ll include in the study.
With the use scenarios in place, official study design begins. Expect lots of interaction and draft reviews of all planning and test materials. Two key outputs will be:
-
A test protocol, and
-
A moderator guide for each user group being tested.
This is a good time for your internal team to ensure they have the required hardware and training materials ready.
Conduct a dry run of the test according to the protocol and update it if necessary. Ideally, this would involve an actual user that meets the screener criteria. But it’s not mandatory since you’re not taking any data for a report just yet.
Step 2 – Testing Day
Spending ample time in the “prepare” phase is crucial. It ensures the right data will be collected during your human factors validation study. All to say, don’t rush to this “testing” phase.
At MPE (Formerly Mindflow Design) , our experience includes usability testing in many geographic locations and types of facilities. Sometimes, a specific environment is needed. Other times, you may need to take the testing to a city with a higher population of a specific surgeon, for example. No matter what, make sure you leave adequate time for orientation to the space and set up.
Your investment in a recruiting agency starts to pay off when each of your participants trickles into the right place at the right time. In the case of two user groups, that could be 30 people over a one to two week period!
What to Expect During Your Human Factors Validation Study
The usability evaluations themselves typically consist of three tasks:
-
First, the participant is trained on how to use your product. Note, there are cases when participants are not trained before using the product. For example, for some CLIA-waived products, participants may receive the Operator Manual and other instructional material to review if they choose, but are not provided with a dedicated training session prior to use. The decision goes back to the intended use statements and if consistent training is expected and enforced for all users. If training is included, this may be done by a rep from your company or through a Train-the-Trainer. The point is that training should be as similar as possible to the real-world training users would receive before actual commercial operation. Remember, you’re trying to simulate an authentic experience with your medical device.
-
Second, the participant must wait a specific amount of time between completion of training and the beginning of testing. This memory decay period should be at least one hour — but could be much more. Ideally, all participants would wait the same amount of time if being trained individually.
-
Finally, the moderator runs each participant individually through the task-based evaluation according to the test protocol. Test duration is determined by the number of tasks the participant is asked to complete and the time it takes to complete. This could be as little as one hour per participant. It’s a good idea to video record each session for review if necessary. Once complete, the participant receives payment and the team scurries to get ready for the next participant, who should be in the waiting room.
Non-study personnel are welcome to observe the sessions if a professional test facility like MPE (Formerly Mindflow Design) ’s Living Lab is used — a facility where the observers are separated by a one-way mirror. During the sessions, emphasis is placed on the participant performing the activities as they would in the real world. Consequently, think-aloud protocol is not used, and test personnel do not intervene.
Step 3 – Data Analysis and Reporting
The human factors moderator is usually the one responsible for data analysis. If a second observer was used to log data, the two will consult and compare notes.
Keep in mind that we always include a second — if not a third and fourth — observer. Testing days are long. You can’t rely on the continuous vigilance of a single observer; that person needs breaks. Having more than one set of eyes on users is therefore a best practice. What’s more, in the event of a discrepancy or failed task, you can pull from multiple people’s perspectives before coming to a more well-rounded consensus.
Reviewing the Data for Use Errors
A lot of time, money and effort have been invested in summative testing to this point. It’s natural to become impatient and start pushing your team (or your external partner) to get you the report. We urge you to resist this impulse and support the team by allowing the time needed to draw accurate conclusions and be thorough in reporting. This process is typically three weeks.
Why does it take about three weeks? The human factors engineering experts have to discuss observations, as mentioned. Per the FDA, they also categorize observations as follows:
-
Use Error – User action or lack of action that was different from that expected by the manufacturer and caused a result that was different from what the user expected and was not caused solely by device failure. This can be further broken down into:
-
Safety-critical – Use error that has the potential of resulting in a harmful outcome to either the user or the patient receiving care
-
Non-critical – Use error that may not lead to a serious harm but could impact device ease of use
-
-
Close calls – An instance in which a user has difficulty or makes a use error, but the user takes an action to “recover” and prevents the use error from occurring
-
Difficulties – any action that makes the user experience more difficult
-
Other observations of interest – notable aspects of the study that don’t fit into an above category
Once the team categorizes the use errors, they have to identify the root cause — the underlying issue that led to the observed use error — for each.
Writing the Final Report
Finally, they write up the report. And it’s not as simple as calling attention to the safety-critical use errors.
For example, even difficulties offer valuable insight into your medical device. A pattern of difficulties suggests a larger usability issue you can make a plan to mitigate in the event the FDA flags it as problematic.
A Stronger Case for FDA Approval Is Now Yours
Once the report is complete, the research lead should present it to your design team. Why? Because teammates who likely have been working on the design for years should be able to hear validation of successful acceptance by a physician or caregiver.
If the device didn’t perform as well as expected, the designers need to hear that too. That way, they have the chance to question the research lead directly. And decide the best next steps.
Your summative report will become part of your usability engineering file, which in turn becomes part of your FDA filing. Of no small consequence, the summative report will strengthen your case for FDA approval. And most importantly, you’ll have proven that you designed the right product for the people who matter most.
