Test Early, Test Often

How to take better advantage of your most valuable resource: users

Brian McKenna
Jun 24, 2019 · 5 min read
Photo: Paper Boat Creative/Getty Images

AA key hallmark of user experience design is talking with users and aligning with their goals and needs. Most of the focus, however, is in two key phases: conversations up front to discover product needs (discovery and generative research) and another round once a concept has been laid out (usability testing). These research phases are, of course, essential to product development. Without each of these in your process, you’re just doing UX theater, not actual design. But your interactions with users need not be confined to just these two steps. Every single point in your process and every artifact you produce can — and should — be tested and validated with users.

If you’ve successfully convinced your leadership that talking with users is valuable, perhaps you can begin to convince them to slowly expand the number of places in the design process where you can engage users. You can test literally every part of the design process and I want to highlight a few key places you may not have considered.

Research plan

It’s very tempting, especially in a world with limited research budgets, to build out a research plan and run through it with multiple participants all in one short burst. This gives you a whole host of data to sort through, analyze, and synthesize. It saves time, money, and energy.

But each research session you have with someone should shape your view of the problem space. Each should open new avenues to pursue, contradictions to explore, and build upon or tear down your current hypotheses.

Your research is about understanding the problem. You should be taking advantage of the knowledge gained in each user session to revise what you plan on discussing in your follow-up sessions or where you are focusing your attention for your observations.

Ideally, you should take some time between each session to revise your research plan. Your plan should not be a static artifact. Expand your line of questions and observations as necessary. Remove lines of inquiry if they’re not revealing anything useful. Figure out where you need to spend more or less time. Your participants’ time is valuable. Your time as a researcher is valuable. Respect this time and focus on refining research items that will provide the most understanding of the problem.

Each [research session] should open new avenues to pursue, contradictions to explore, and build upon or tear down your current hypotheses.

Analysis

You’ve done your research. You’ve completed your synthesis. You’ve built your personas and constructed your journey maps (or whatever your analysis artifact of choice is). Usually, the next step after this is to share and review these with relevant stakeholders. You walk through them with the product owner. You discuss what you learned with the leadership team. These are important in order for decisions to be made.

Our analysis artifacts help capture what we understand about the user’s problem space. We should validate that with users, but often don’t. (Photo credit)

But for some reason, we rarely review these things with users. We give the people we are supposedly modeling no input on whether we have actually captured their worlds correctly.

But why? Is it because we think our analysis artifacts are too difficult for people to understand?

Showing users your analysis documents is a reasonable option to at least consider. If you can get to a place where you are doing continual research, this analysis artifact should be the central way to fill in the gaps in your understanding. You can ask pointed questions and challenge the conclusions you’ve drawn by checking with the source of those conclusions: the users.

If you are confident, you can introduce and explain the documents to the users before showing them the details they contain. If not, keep the artifacts to yourself but use them to guide your feedback session.

If the goal is to build a product that solves the right problem, there is no better way to validate this than to show your understanding of the problem to the users before you even design for it.

Test plan

If you are testing your product with users, you should plan to throw the first few sessions away. Like anything else, you shouldn’t expect to make a perfect test plan on the first pass. Yes, internal reviews with others on your team will help smooth out some edges, placing you in a fairly good place once you begin testing with users. Yet going live with users for the first time always seems to help uncover problems with the execution of your test plan.

Perhaps your language isn’t clear. Maybe tasks are taking longer to complete than you thought. Maybe they are taking less time than you expected and you can squeeze a few more tasks in for data collection. Sometimes, your pilot participants will take you down a road you hadn’t expected. In multiple cases, I’ve started out with my pilot participants only to realize that my data collection approach wasn’t set up correctly and I needed to revise.

These are all ways that users can provide feedback on your test plan. And you will stumble on most of these issues within one or two participants. But you have to understand this need and build out time (and find extra test participants) to accommodate the troubles that typically come with running product tests with users.

WeWe all know that learning from our users is extremely important. The more we can expand that footprint to all aspects of our work, the better off we will be.

User feedback and testing are not stages in our process. They are inherent aspects, embedded in everything we do. These are our feedback mechanisms to make sure we deliver on what we are responsible for and remain on the right path. Why wouldn’t you employ that in every facet that you can?

Modus

Helping designers thrive.

Brian McKenna

Written by

Designer. Customer Experience Director. Been at this for 15 years. Live in Pittsburgh, but will always be a Chicago guy. Go Cubs! On twitter: @bkenna1

Modus

Modus

Helping designers thrive. A Medium publication about UX/UI design.

Brian McKenna

Written by

Designer. Customer Experience Director. Been at this for 15 years. Live in Pittsburgh, but will always be a Chicago guy. Go Cubs! On twitter: @bkenna1

Modus

Modus

Helping designers thrive. A Medium publication about UX/UI design.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store