Summative Usability Testing: Why and How to Do It
The business of introducing new medical devices requires us to address enormous technical challenges that require plenty of hard work. As we strive to meet these challenges, the FDA reminds us not to lose sight of people’s needs through a framework focused on keeping users, rather than technology, at the heart of the design process.
Summative usability testing is a tool we use to assure that all-important “user first” perspective. It summarizes the accomplishments of the design process by testing a production-like system at its conclusion. This testing method validates how usable and safe your product is through defined measurements, reducing any risk of costly surprises.
Such testing can be either fairly straightforward or complicated depending on the potential risk a device might cause patients or operators. In either case, it will go much smoother if you begin planning early in development. If you didn’t plan early and are much farther along, don’t worry – you’re not alone and you can still complete successful summative testing.
Here are the three basic steps:
Step 1 – Preparation and Planning
Choose a firm that specializes usability testing of medical devices or engage with your internal human factors resource. This isn’t an activity for one of your marketers to tackle because it requires a very specific skill set and testing procedure.
Assign the right core-team of internal people and start with an official kickoff workshop. Outside of the human factors experts, we recommend including at least one person from engineering, marketing, regulatory affairs, training and a point person to manage the effort.
Determine the user group(s) that need to be tested and create a participant screener for each one. Get them done as soon as possible so you can get the three to four week recruiting process started.
The FDA expects you to test at least 15 people for each user group, but we recommend recruiting an additional three incase people dropout. Finding and scheduling participants is the single most difficult task in summative testing, so we highly recommend using a recruiting agency. The more specialized your users are, the harder it will be to recruit them and the more you’ll need to pay them for their time.
The human factors lead will design the study in close collaboration with the core-team. Expect lots of interaction and draft reviews of all planning and test materials. Two of the key outputs will be a test protocol and a moderator guide for each user group being tested. This is a good time for your internal team to ensure that they have ready the required hardware, instructions for use and training materials.
Conduct a dry run of the test according to the protocol and update it if necessary. Ideally this would involve an actual user that meets the screener criteria, but it’s not mandatory because you’re not actually taking any data for a report.
Step 2 – Testing Day
Our experience includes usability testing in many geographic locations and types of facilities. Sometimes a specific environment is needed. Other times, you may need to take the testing to a city that has a higher population of a specific surgeon, for example. In either case, make sure you leave adequate time for orientation to the space and set-up.
All the money you spent with the recruiting agency starts to pay off when each of your participants start to trickle in at the right place, at precisely the right time. In the case of two user groups that could be 30 people over a one to two week period.
The usability evaluations themselves typically consist of three tasks:
First, the participant should be trained on how to use the device. The training should be as similar as possible to actual real-world training they would receive before commercial use. Remember, you’re trying to simulate a real-world situation.
Second, the participant must wait a specific amount of time between completion of training and the beginning of testing. This memory decay period should be at least one hour but could be much more. Ideally, all participants would wait the same amount of time if being trained individually.
Finally, the moderator runs each participant individually through the task-based evaluation according to the test protocol. Test duration is determined by the number of tasks the participant is asked to complete and the time completion takes. This could be as little as one hour per participant. It’s a good idea to video record each session for review if necessary. Once complete, the participant receives a payment check and the team scurries to get ready for the next participant who should be in the waiting room.
Step 3 – Data Analysis and Reporting
The human factors moderator is usually the one responsible for data analysis. If a second observer was used to log data, the two will consult and compare notes. In the event of a discrepancy or failed task, they might review the video recording to form their final opinion on the matter.
A lot of time, money, and effort has been invested in summative testing to this point. It’s natural to become impatient and start pushing your team to get you the report. We would urge you to resist this impulse and support the team by allowing the time needed to draw accurate conclusions and be thorough in reporting.
Summing It All Up
Once the report is complete, the research lead should present it to your design team. Why? Because team-mates who likely have been working on the design for years should be able to hear validation of successful acceptance by a physician or caregiver. It’s their moment to shine and they deserve it. If the device didn’t perform as well as expected, they need to hear that too and have the chance to question the research lead directly.
Your summative report will become part of your Usability Engineering File which in turn becomes part of your FDA filing. Of no small consequence, the summative report will strengthen your case for FDA approval. And most importantly, you’ll have proven that you designed the right product for the people who matter most.