What is a Formative Usability Evaluation? (And Why Does Your Medical Device Need It?)
The most successful medical devices aren’t just safe and effective. They also make their end users’ lives easier. Unfortunately, an excellent user experience (UX) is far from inevitable. In fact, a great UX sets medical devices apart precisely because designing a truly user-centric experience is so challenging.
Yet the key to creating these experiences isn’t some deeply held secret. It’s simple: You must get to know your users and listen to their feedback about your product as you design and develop it. In short, you need to understand and leverage formative testing.
So what exactly is formative testing? How does it work? And when should you employ it? Here’s what you need to know before you start developing your next medical device.
What is a Formative Evaluation in Medical Device Development?
A formative evaluation is an activity that allows medical device development teams to learn more about the products they are developing, specifically through the lens of their end users. Generally speaking, a formative evaluation is an event in which end users (and other stakeholders) evaluate, interact with, and respond to a proposed experience, concept, or prototype. There’s no set structure to formative testing. However, you may find that evaluations conducted in the intended environment (such as a hospital, clinic, or home) could yield better results.
While formative evaluations are structurally flexible, the underlying purpose is always the same. That is, to put user needs and preferences at the center of a medical device design team’s decisions from the earliest stages of the development process. For that reason, formative evaluations mostly focus on qualitative findings rather than more quantitative pass/fail performance “tests.”
The FDA highly recommends that you conduct formative evaluations and would like to see the evidence in your submission. But the real benefit is the opportunity to more perfectly tailor your device to its users’ needs. But the benefits you extract from it depend on whether you treat it like just another box to check — or a welcome opportunity to more perfectly tailor your device to your users’ needs.
Ideally, you should conduct formative evaluations at various points throughout the development process. And, they can be especially useful as part of the discovery, qualification, and verification phases.
In general, formative evaluations range from more casual to more rigorous depending on where you are in the product design process. During the earliest stages, formative evaluations enable you to learn more about the product you intend to build. It can help you define the use case, identify hazards and risks, and key in on conceptual solutions for your medical device. Later, as you solidify your concept, you can use formative evaluations to ensure that your design’s level of refinement is appropriate for the next phase of development.
The Benefits of Formative Usability Evaluations
By engaging in a robust program of formative evaluations, you can:
-
Reduce the risk of developing the wrong product
-
Reduce the risk of developing a hazardous product
-
Provide insights to develop solutions that support users’ needs and improve outcomes
-
Improve your medical device’s overall user experience and reduce errors
-
Demonstrate and qualify that a solution is ready to advance to the next (more costly) phase of development
The Four Phases of Formative Evaluations in Medical Product Development
You should stage out all of your planned formative evaluation activities at the outset of your medical device project. These activities should be encapsulated in your human factors engineering plan, a roadmap that includes all formative and summative usability and human factors activities. The FDA doesn’t mandate when or how often to include formative evaluations. However, user-centered design focused organizations know to weave in formative testing early and often. Typically, formative evaluations can be broken out into four distinct phases with slightly different activities and goals. These include:
Discovery testing: what are we building?
In the earliest phases of a new medical device project, formative evaluations are the key to uncovering what you need to build. Of course, you likely already have a pretty strong idea of what your medical device will be — what it will do and even how it might function and look. But, formative evaluations will help you confirm your suspicions, or in the event that they are wrong, course correct early.
Market research isn’t the same thing as user research. Your marketing and business teams may be sure that you ought to bring X, Y, or Z to market. But you should never set your design course without first checking in with your users. During the discovery phase, formative testing is the easiest and most effective way to unearth user feedback.
Formative evaluations will be more informal in this early stage. For example, you might prepare a loose discussion guide and invite a group of core users and key opinion leaders (KOLs) to a roundtable discussion. You should also plan to conduct in-person observational research, which is also called contextual inquiry. For this activity, your team observes your end users in action as they complete relevant tasks in their working environment. Contextual inquiry may include light interviewing, but researchers should mostly stay out of the way and watch as users do their work.
There are a number of factors to consider during observation. These include the environment (light, space, noise level, etc.), the technicality of individual interactions or tasks, the number of users involved, and their level of expertise.
These early formative evaluations in combination with other benchmarking activities should give your team a more precise high-level understanding of what, exactly, you should be developing. You can then start translating your user needs into actionable design inputs.
Configuration evaluations: how should our device be composed?
Next, your team could conduct formative evaluations to determine the best overall product configuration. This is fundamental to laying a solid foundation and moving in the right direction early on.
Use a discussion guide to present users with two or more low-fidelity configuration prototypes. At this stage, the prototypes should be more conceptual than detailed. The idea is to present concepts that have just enough specificity to elicit discussion of your device’s configuration. For example, you may want to get your users’ feedback on the placement and number of handles on your device. The handles you include in your prototypes should offer affordance for lifting or grasping, but be devoid of any design details that could sidetrack the conversation.
Allow your participants to evaluate and compare your rough prototypes with the goal of understanding their preferences related to your device’s high-level configuration. Should the handles be located on the top or the side? Should the screen be oriented in landscape or portrait mode? Should the Input/Output panel be positioned here or there?
Use the feedback you receive in configuration evaluations to confirm the essential design inputs of your device’s design before making more detailed design decisions.
Risks and hazards testing: how can we minimize hazards and design for optimum safety?
Formative testing can also be used to minimize hazards and risks. The FDA’s guidance on human factors and IEC’s standard 62366-1 details the requirements and best practices of formative testing as it relates to risk assessment. This should be considered alongside ISO-14971, which describes the FDA’s requirements for hazard risk management and risk analyses within medical device development.
Your human factors team should start by identifying and documenting the risks that are present within your medical device’s design. Next, you’ll need to evaluate those risks in the context of a user testing environment.
Human factors testing: are we on the right track?
As you hone in on an increasingly high-fidelity prototype, you’ll once again leverage formative usability evaluations — this time to confirm that your detailed designs are headed in the right direction.
We recommend planning and conducting a more structured formative evaluation in a simulated or actual environment. These late-stage formative evaluations may also include some quantitative measurements, such as task completion rates and times. The goal at this point is to put the finishing touches on your design prior to conducting a rigorous summative evaluation test.
Medical device companies must navigate a plethora of FDA regulations in order to prove that their products are safe and effective. Meeting these requirements isn’t a differentiator; it’s simply the price of admission in taking a medical device to market. The best way to differentiate your medical device? Make it as user-friendly as possible. This means formative evaluations aren’t just a recommended activity by the FDA to cross off your list. It’s an incredibly valuable tool that can meaningfully shape your medical device’s design by revealing much-needed insights into the why behind your product.
