Webinar: How to avoid the common mistakes of formative evaluations to meet user needs

Webinar Title: How to avoid the common mistakes of formative evaluations to meet user needs

Chris Ross (speaking):

Hello everyone and thanks for attending our webinar today – How to avoid the common mistakes of formative evaluations to meet user needs

I’m Chris Ross a principle of MindFlow Design. I’ve been designing Healthcare related products for just over 20 years now.

We started our company back in 2007 designing and developing healthcare products for medical, life sciences, and home health companies. They range from startups to the industry’s largest. We help them from concept to complete manufacturing specifications. This includes user research, industrial design, engineering, UX/UI design, and human factors usability evaluations, both formative and summative testing.

Over the years we’ve continually noticed companies struggle to identify user needs and get effective feedback.

In particular the cycles of user feedback a design team needs to assure they’re designing the right product.

We believe so strongly in our topic today, that we built a separate 2,500 ft2 facility in our building called the Living Lab User Research Testing Facility that we use and even rent out.

Today I hope to demystify the process of getting user feedback on your concepts and prototypes early and often during development.

The purpose of this webinar is to make you aware of How to avoid the common mistakes of formative evaluations, so you can raise your organization’s user testing game.

This webinar will equip you to look beyond formative usability testing as an FDA box to be checked. You will gain a new appreciation for its value in discovering how your product can best meet user needs, or even discovering needs that might be met by entirely new products.

We have about 1 hour together today, including time for some questions and answers at the end. Let’s get started.

Have you ever had the feeling a product was designed just for you? It’s a great feeling, right?

Some company out there cared enough to make your life easier. Empower you. Build your confidence. Or, perhaps even put a smile on your face.

If you’re in healthcare that can be especially gratifying because you are helping people in need.

Seasoned medical product developers know that an excited customer reaction doesn’t happen by accident. On the contrary, it takes a lot of work — the result of a goal that must be established early on and adhered to over the project duration.

Wouldn’t it be great if the reaction you got every time from your customers and the people who use your product was “WOW, it feels like they designed this just for me!” or “I didn’t even know I needed one of these?”

This result requires putting customers at the center of your medical product design process but that is actually harder than it sounds. In fact, we’ve found that most companies don’t do it very well, and we’re not alone. The FDA noticed also.

In 2011 they published a draft guidance to tell medical device makers how to do it effectively. Then they released the final document in 2016.

If you haven’t already, I recommend downloading a copy from the FDA’s website and taking a look.

There are two points in the document that I’d like to share now

This is the very first sentence in the document:

FDA has developed this guidance document…to maximize the likelihood that new medical devices will be safe and effective for the intended users, uses and use environments.”

The FDA’s primary focus and really all they care about are safety and effectiveness.

The second point is:

“FDA’s guidance documents, including this one, do not establish legally enforceable responsibilities. Instead, guidance documents describe the Agency’s current thinking on a topic and should be viewed only as recommendations unless specific regulatory or statutory requirements are cited.”

This is not a standard or requirement. In fact the header on all 49 pages says “Contains Nonbinding Recommendations”.

I believe this undermines the importance of having a solid human factors engineering program. The reality is that when money and time are tight during a project certain human factors initiatives get pared back or skipped all together.

The formative evaluations we’ll discuss today are one of the common victims.

What are formative tests or evaluations anyway? Let’s look at a simple graphic before we dig into the FDA’s definition.

Section 1: What is a formative evaluation?

This is the waterfall diagram from 60601 that depicts the design controls process. It’s the foundation of your medical device development process. It should look familiar to anyone who works in medical device development.

So the 10,000 foot explanation is Figure out what your users need , translate the needs into inputs and requirements, design a product to those requirements, create specifications as outputs, verify that you’ve created the product according to the specifications, make your device and validate that your product meets the original user needs. OH – and don’t forget to conduct formal design reviews as gates between those steps.

Formative evaluations are part of the verification process.

Keep that visual in mind as we review the FDA’s definition of formative testing

This is how the FDA defines it in Section 3.3 Formative evaluation – from the FDA Guidance: Applying HF and Usability Engineering to Medial Devices

“Process of assessing, at one or more stages during the device development process, a user interface or user interactions with the user interface to identify the interface’s strengths and weaknesses and to identify potential use errors that would or could result in harm to the patient or user.”

This is a great definition, but it implies that a device simply needs to be safe and effective to satisfy peoples’ needs.

Here’s a visual of what that would look like.

Remember the “WOW” factor from the first slide? You’re not going to get there by solely meeting the FDA’s two aims.

We believe the FDA’s definition is a bit short sighted and can be misleading to medical device companies. It’s easy to get so focused on meeting the FDA’s recommendations and requirements that we lose sight of the human side: Emotions, Feelings, Confidence, Love, Certainty, Delight

Here’s what we think that should look like…

Don’t we want to delight people, not just keep them safe?

So in terms the FDA might use, we want to mitigate against the opposite of “delight”. Let’s use the word “Frustration”.

If we were to rewrite the FDA’s definition it might look like this

“Process of assessing, at one or more stages during the device development process, a user interface or user interactions with the user interface to identify the interface’s strengths and weaknesses and to identify potential use errors that would or could result in harm OR FRUSTRATION to the patient or user.”

It adds people’s “feelings” into the equation which we believe are just as important to the marketplace success of a product as mitigating risk of harm or just getting the job done.

Now that we have a better understanding of what a formative evaluation is, let’s see where they take place during the design and development process

It’s important to interact with users throughout the entire development journey. And that will probably look different for each project. But there are generally three points during the design process that we interact with users:

At the beginning – We conduct user research to uncover needs and translate them into requirements that will drive the design of the product.

This requires going out into the field and observing people and interviewing them in context of doing what it is they do. That might mean being a fly on the wall during surgery or visiting people in their homes to better understand their daily ritual for taking medications.

Most design research should be conducted at the beginning of the development process.

In the middle – This is where most of your formative evaluations will take place.

The main purpose is to determine if proposed design solutions meet the requirements and if they do not, modify the requirements OR the design accordingly.

They are your main feedback loops with users, taking place throughout the entire design & development process. Multiple cycles of design – evaluate – iterate.

At the end to Validate that the final design is safe and meets the original user needs.

The FDA calls this Summative usability testing and it takes place after the design is complete. If you follow a solid HFE plan and conducted effective formative evaluations you shouldn’t encounter any surprises at this point.

Regardless of when we interact with them, it’s important to remember that we’re not actually evaluating or testing users. They are evaluating our ideas, concepts, and designs. What we hand them and how we ask them to evaluate it is important.

Let’s refocus our attention back on formative testing and see what that looks like.

Since users will evaluate your designs multiple times during development the prototypes you use early will have much less detail than the later ones. In the same way, what you seek to learn from those prototypes early on is much broader and more detailed later. And, the rigor of your testing will follow suit.

Let’s look at three prototypes used during the development of an acute care ventilator.

The first one is made of cardboard with a conceptual graphical user interface image glued to it. We wanted feedback and first impressions of the overall configuration and size.

The 2nd is a foam model with a little more detail including the proposed weight of about 125 lbs. It can be dismantled by users into three parts during user evaluations to simulate that proposed feature.

The last image is an appearance prototype that looks and feels real but is also a solid block of foam with no inner functionality.

However, all user interaction points – filter replacement, air and electrical connectivity, battery ejection and replacement are fully functional. There’s even a fully functional Touchscreen LCD run by an external laptop to include evaluation of the graphical user interface in context of the physical interface.

Here’s another example of prototypes we made while developing this oxygen concentrator. This home-health product is used by people with breathing disorders 24/7 throughout their daily lives.

We had users evaluate the first set of three very early on to determine which form factor would fit best into their daily lives.

The one in the middle is a foam model at the correct weight. Since this is a portable product used 24/7 it was important to have users interact with it in their homes and automobiles to make sure it worked for them.

The last one is a urethane casting and is fully functional. Over 15 of them were made to get user and customer feedback across the entire country.

It’s especially important to follow a sound HFE plan and document all your studies for home-health products. The FDA is carefully watching the migration of increasingly sophisticated medical devices exiting hospitals and entering our homes.

Device developers can often times get away with not having certain aspects of their hospital products tested by users. The rational is that the primary user is often highly trained, familiar with similar products or procedures, and accustomed to an industry standard protocol for that type of product.

Companies developing home-use products can’t use that type of rationale and should consider this when planning formative evaluations.

Let’s look at the three steps of formative testing.

It requires Planning, Testing, and Reporting. Let’s take a quick look at each of them.

Step 1 is Planning

I’m going to walk through six questions you should answer during the planning step. The answers to these questions will help you build a solid plan.

1. What do you want to learn?
2. What are you evaluating?

Earlier evaluations might be testing higher level propositions, ideas around system configuration or workflows, and use lower fidelity prototypes. Later evaluations tend to be more focused on the details. Specific interactions, user interface details, specific tasks, risk..

3. Who should evaluate your concepts or prototypes?

The short answer; Representative users! A sample of the population for each user group. You’ll need to be specific and write it down. You’ll need to create a screener for each group.

(discuss screener example)

4. How do we find participants?

We recommend using a professional recruiting agency to recruit and schedule participants. They have existing databases. It’s tempting to do it yourself to try and save money, but the risks are great. We recommend against it. Those who do it themselves usually end up paying more in the end, delaying the study, and settling for who they can find over the right candidate. Testing and recruiting is expensive, include it in your initial budgets and plan accordingly. Agencies usually request between 2-4 weeks to recruit and schedule.

(Discuss recruiting quote example)

5. How many people should you recruit?

62366-2 states that many usability problems can be discovered in qualitative studies with sample sizes in the range of 5-10 participants per distinct, primary user type or group.

This graph is telling us that less than 5 people and the probability is high that we’ll miss some important findings. More than 10 and the probability is high that you’ll just start seeing more of the same errors and nothing new.

For example, if both doctors and nurses use your device you might want to recruit and test 8 of each or a total of 16 people.

6. How do we conduct the actual evaluations?

All testing requires a written test plan and protocol. Here’s the table of contents of an example.

Step 2: The second step is the testing itself

The testing environment and location could range from in-home, to medical facility, to mocked up environment, to a conference center at a hotel.

This is an early formative evaluation conducted in an office. It wasn’t necessary to use an actual diagnostic laboratory for what we wanted to learn about this prototype.

This is an early formative evaluation conducted at a hotel facility. In this case it wasn’t necessary to go to people’s homes for the evaluations.

This is a simulated clinical environment at our Living Lab testing facility. It can be set up as a hospital room, doctor’s office, or diagnostic laboratory. This suite is comprised of two rooms. The top photo is the testing room. It has cameras on the ceiling and a one-way mirror. The bottom photo is of the adjacent viewing room where the design team can view their products being evaluated firsthand.

This suite is set up as a home living room, but it can also be converted into a bedroom, kitchen, or other home environment as needed.

There’s no right or wrong environment. It just needs to be appropriate for the type of testing outlined in the plan. Keep in mind this may mean going to multiple geographic locations or even countries.

It’s a great opportunity for key team members to see user’s interacting with concepts or prototypes firsthand. Either by video recording, observing live, or watching behind a one-way mirror.

Step 3: Let’s take a look at the last step: Reporting

Don’t lose sight of the fact that the primary reason for a formative evaluation is to gather feedback that will be used to make the design better. The secondary reason is to meet the FDA’s expectations.

Just like the plan and testing itself, the reporting will likely be looser early on and more stringent later in the development process.

For example an early report from early testing might have data and findings expressed in paragraph format. Later on in the design process they might more data driven and expressed like this with more data.

In either case, we always recommend having the research team present the finding to the development team so they can really dig in with questions.

Doing them early and often will help you develop a better product.

Be diligent about your record keeping for all 3 steps. This will be submitted to the FDA in your DHF.

Your development program will benefit in many other ways from formative evaluations. We’ve covered the most important and obvious ones:

First Benefit is getting FDA approval or clearance AND the second benefit is making your users and customers happy.

There are three more benefits I’d like to mention:

Benefit: Promotes deeper level of internal team engagement

Watching a user interact with your concepts and prototypes first-hand makes it personal.

It builds empathy and a deep seeded desire to help your users.

If you give everyone on your design team this same experience their conversations will shift from – How do, we improve the design? To – How do we improve the experience?

How do we make Dr. Mary’s day at work even better?

How do we help little Johnny live a normal life after a diabetes diagnosis?

How do we help Karen regain her confidence after a slip and fall in her home?

That perspective will begin to trickle down into everything they do individually or as a team.

Benefit : Another benefit we see is the onboarding of new team members

The reality in business is that people come and go from companies and projects. We recommend inviting new employees or new members of the project team to watch formative evaluations.

Again, this makes their mission personal and builds empathy. It can give them a deeper sense of purpose so when they ask themselves, “Why am I going to work today?, they are reminded that it’s to help Mary, Johnny, or Karen

Watching live isn’t the only option. You can watch recordings instead.

For example earlier this year we hired a new Industrial Designer who needed to get up to speed on a project quickly. During the first day of watching testing videos she was already adding meaningful and insightful information to the user experience map for that project.

Another tip is to show snippets of formative testing videos to executives who are often distant from the day to day project activities. If you are the executive, ask to see a highlight real. It will change your perspective and build empathy for both your customers and your team.

Benefit: Uncover gaps and opportunities

And the fifth and last benefit?

Having intimate 1 on 1 interactions with end users like using a crystal ball

Who wouldn’t want to have a crystal ball? To know the future before it happens.

Formative evaluations will uncover gaps and opportunities. What people really like about your solutions and what they don’t. This will help you be proactive verses reactive.

Gaps that are not addressed early can lead to big problems later like:

• Service calls or returns
• Negative brand reputation
• Confusion in the marketplace
• Not receiving FDA clearance, and so on.

Opportunities addressed early can:

• Be leveraged to deliver a “Wow” factor
• Increase the overall value proposition of your product, service, or system
• Increase brand equity and reputation

Unfortunately companies who skip or do ineffective evaluations won’t enjoy any of these benefits. Let’s look at the 6 most common reasons they are skipped so we can avoid it.

Six reasons medical companies skip user testing

Reason 1 – “It’s just the next generation of our current product.”

We’ll ask our sales people instead of users since they are boots on the ground and hear directly from customers about our current product.

This is a good place to start, and you should always mine your sales folks for this type of information AND keep a recorded document of it. But, it’s not a substitute for your human factors or design team interacting directly with the users of your product.

The assumption is not much will change from our current product. But in reality, maybe much more SHOULD change but you just don’t know. Having users interact with concepts for the new idea becomes a foundation for a conversation with them. That deeper conversation can then begin to surface insights or nuggets that weren’t realized before.

I recommend beginning every new development project with design research. Send research teams out into the field to watch and interview people using your current or competitor’s products. Don’t disclose what company you work for to get unbiased information. Stealth-like, unbiased research might be better suited for a third-party firm. We do this all the time for our clients.

Regardless of who conducts it, this research will likely lead to fresh new ideas that need to be vetted with formative evaluations.

Reason 2 – “We know our customers. We’re the industry leader.”

This is the common big company mindset that causes companies to stop looking for new ideas and innovations. It’s the opposite mindset of the 2 guys in a garage pursuing a disruptive idea spawned after a trip to the hospital.

Once companies start to feel like they’ve “heard it all” they are no longer listening. No longer sensitive to the pain-points customers are experiencing. In fact the worse problem is the professionals using hospital equipment and devices are so use to their daily routines and putting up with whatever the hospital provides for them to use, that they don’t even know to complain anymore. So use to their daily routine that they can’t even tell the device makers the problems to solve.

Intimate interaction during concept evaluations with users allows the trained eyes and ears of a human factors expert to note frustrations, workarounds, and again go into a deeper dive with the users to uncover unmet needs that are made invisible and driven to the subconscious.

Reason 3 – “We’ll conduct VOC right before it launches.”

VOC stands for voice of the customer and is a process/technique used by many companies for staying in touch with their customers and users. It’s a great idea and can be highly effective depending on how it’s used or the particular process the device maker follows. Most VOC programs are customized and tailored to the needs of a specific company. The problem is that VOC becomes a label with different meaning to different people.

You might find yourself reassuring an inquiring executive that all VOC has been completed. That the product has all the features and benefits that are required. In terms of user needs, the VOC data is only as accurate and true, as the level and type of user interaction that took place during design and development.

Many companies will take out the more final design during the home stretch of development when it’s actually too late to change. What they’re really doing is “validating” the design they’ve already created and, in many cases, simply looking to get enough positive opinions about the new product that they can say “9 out of 10” doctors prefer product x. In that case the VOC is really to support claims that can be used during sales and marketing.

Formative evaluations are for the design team not marketing. They are not to gather voice of the customer. They are to engage in a deep and meaningful way with users, to gain feedback that the design team uses to improve the design and make critical decisions about design direction.

Don’t listen to what they say, watch what they do.

Reason 4 – “We have clinicians on staff who will look at it.”

Many device makers employ physicians that help guide the development of their products. They’re clinical input and fulltime dedication and involvement are surely needed and are indispensable. However, there’s no substitute for the outside objective feedback.

An outsider has the advantage of purely judging the design proposals for what they are. No incentive, no past knowledge to clouding their vision.

Sometimes the in-house clinical experts have been working at the company and have not actually practiced in many years. Things change, hospitals change, ways of doing things change in the clinical setting, treatments and diagnostics change. It’s important to get feedback from the right people who are currently working under the conditions and who have nothing to gain except a better product to use in the end.

It’s not to say that in-house clinicians shouldn’t be involved, because they should. Just don’t rely on their opinions alone. Make sure to always include a healthy number of users outside of your company.

Reason 5 – “We did do formative evaluations!”

Many companies don’t do them effectively which is one of the main reasons for this webinar today. This one is actually the biggest problem because you think you’ve mitigated a risk and checked it off the list. However, the misinformation could lead to unwarranted design changes or a pivot. Sending your project in the wrong direction without anyone even knowing.

The common culprits are poor planning, bad execution, or evaluations by the wrong type of people which is probably the biggest. We like to say, “Garbage in, garbage out”.

Even if done correctly, many companies don’t do enough of them. The more times you engage in a productive way with users throughout development, early and often, the more tailored your design becomes to meet or exceed their needs.

If the testing was ineffective, you skipped it!

Reason 6 – “It’s not in the budget.”

Formative evaluations are expensive and require planned investment. Lack of budget is a planning problem typically caused by lack of awareness early on for what needs to be done.

A good project diagnostic at the beginning should identify these budget gaps. We’ve implemented proprietary diagnostics into our work to help our clients be proactive rather than reactive.

IEC 62366-2 Application of Usability Engineering to Medical Devices explains what you need to do. During early project planning I recommend adding each of the sections from the guidance as an activity requiring budget, into your Gantt schedule. That will assure that the leadership team is aware of the activities and HOPEFULLY, allocates a budget.

Those are the 6 most common reasons companies skip formative evaluations.

We feel strongly that if you do formative user testing right then everyone benefits.

There’s a lot at stake and We owe it to…

• The patients who expect and deserve the highest quality care.
• The users who expect our products to delight them and simplify their lives.
• The FDA who hold the keys to the gate we all want to pass through.
• Our design teams, so they have the information they need to do their best work.
• Our shareholders and investors. It’s the fiscally responsible thing to do.
• Our customers who are expected to deliver the highest level of healthcare possible.

Let’s recap the main points we went over today.

• We reviewed the FDA’s definition for formative evaluations which focused on safety and effectiveness and expanded it to include delight.
• Formative evaluations should be used early and often, throughout the entire design process
• The types of prototypes you’ll create and test will change throughout a project. The level of fidelity and detail increases as a project progresses.
• Testing requires three steps: Planning, Testing, Reporting
• There are 5 benefits to effective formative evaluations:

1. An all-around better product that people enjoy using.
2. Proving to FDA that your product is safe and effective
3. Promote a deeper level of team engagement
4. They help with onboarding new team members
5. Like having a crystal ball and seeing into the future

• We discussed Six reasons medical companies skip formative evaluations or testing

1. It’s just the next generation of our current product
2. We know what our customers need. We’re the industry leader”
3. We’ll conduct VOC right before it launches.
4. We have clinicians on staff who will look at it and guide our design team.
5. We did do formative evaluations
6. It’s not in the budget

In Closing

I hope I’ve inspired you to think differently about the importance of formative evaluations in your design and development process.

We’ve positioned MindFlow Design and built our expertise to help companies avoid the challenges we discussed today.

We have other tips and articles to share on our website. If you want to learn more about this topic you can go to the search box in the “blog” section and type the word “formative” into the search box.

My contact information is right here. Please feel free to reach out. If there’s any way we can help, I’d love to hear from you.

I’m going to open up the call now for some Q&A. If you have any questions, please type them in the chat box.