Test Early, Test Often
How to take better advantage of your most valuable resource: users
A key hallmark of user experience design is talking with users and aligning with their goals and needs. Most of the focus, however, is in two key phases: conversations up front to discover product needs (discovery and generative research) and another round once a concept has been laid out (usability testing). These research phases are, of course, essential to product development. Without each of these in your process, you’re just doing UX theater, not actual design. But your interactions with users need not be confined to just these two steps. Every single point in your process and every artifact you produce can — and should — be tested and validated with users.
If you’ve successfully convinced your leadership that talking with users is valuable, perhaps you can begin to convince them to slowly expand the number of places in the design process where you can engage users. You can test literally every part of the design process and I want to highlight a few key places you may not have considered.
It’s very tempting, especially in a world with limited research budgets, to build out a research plan and run through it with multiple participants all in one short burst. This gives you a whole host of data to sort through, analyze, and synthesize. It saves time, money, and energy.
But each research session you have with someone should shape your view of the problem space. Each should open new avenues to pursue, contradictions to explore, and build upon or tear down your current hypotheses.
Your research is about understanding the problem. You should be taking advantage of the knowledge gained in each user session to revise what you plan on discussing in your follow-up sessions or where you are focusing your attention for your observations.
Ideally, you should take some time between each session to revise your research plan. Your plan should not be a static artifact. Expand your line of questions and observations as necessary. Remove lines of inquiry if they’re not revealing anything useful. Figure out where you need to spend more or less time. Your participants’ time is valuable. Your time as a researcher is valuable. Respect this time and focus on refining research items that will provide the most understanding of the problem.
Each [research session] should open new avenues to pursue, contradictions to explore, and build upon or tear down your current hypotheses.
You’ve done your research. You’ve completed your synthesis. You’ve built your personas and constructed your journey maps (or whatever your analysis artifact of choice is). Usually, the next step after this is to share and review these with relevant stakeholders. You walk through them with the product owner. You discuss what you learned with the leadership team. These are important in order for decisions to be made.
But for some reason, we rarely review these things with users. We give the people we are supposedly modeling no input on whether we have actually captured their worlds correctly.
But why? Is it because we think our analysis artifacts are too difficult for people to understand?
Showing users your analysis documents is a reasonable option to at least consider. If you can get to a place where you are doing continual research, this analysis artifact should be the central way to fill in the gaps in your understanding. You can ask pointed questions and challenge the conclusions you’ve drawn by checking with the source of those conclusions: the users.
If you are confident, you can introduce and explain the documents to the users before showing them the details they contain. If not, keep the artifacts to yourself but use them to guide your feedback session.
If the goal is to build a product that solves the right problem, there is no better way to validate this than to show your understanding of the problem to the users before you even design for it.
If you are testing your product with users, you should plan to throw the first few sessions away. Like anything else, you shouldn’t expect to make a perfect test plan on the first pass. Yes, internal reviews with others on your team will help smooth out some edges, placing you in a fairly good place once you begin testing with users. Yet going live with users for the first time always seems to help uncover problems with the execution of your test plan.
Perhaps your language isn’t clear. Maybe tasks are taking longer to complete than you thought. Maybe they are taking less time than you expected and you can squeeze a few more tasks in for data collection. Sometimes, your pilot participants will take you down a road you hadn’t expected. In multiple cases, I’ve started out with my pilot participants only to realize that my data collection approach wasn’t set up correctly and I needed to revise.
These are all ways that users can provide feedback on your test plan. And you will stumble on most of these issues within one or two participants. But you have to understand this need and build out time (and find extra test participants) to accommodate the troubles that typically come with running product tests with users.
We all know that learning from our users is extremely important. The more we can expand that footprint to all aspects of our work, the better off we will be.
User feedback and testing are not stages in our process. They are inherent aspects, embedded in everything we do. These are our feedback mechanisms to make sure we deliver on what we are responsible for and remain on the right path. Why wouldn’t you employ that in every facet that you can?