How to Do User Testing on a Budget
A beginner’s guide to running your first user testing session for $0
User testing is expensive. Designers, researchers, product managers, and developers are often tasked with performing user testing on limited budgets. As a designer, I’ve worked on numerous projects under similar constraints and I’d like to share some tips and tricks I’ve learned throughout the years.
Though I’ll focus on testing methods for digital products, the general concepts can apply to services or physical products. The goal is to show ideas to users early to evaluate, iterate, and validate assumptions.
What is user testing?
User testing, also known as usability testing, is a process that identifies issues with a product by testing it with real users. User testing has been around for a while; in World War II it was used to improve military equipment.
Steve Krug’s Don’t Make Me Think further popularized user testing across the industry. Today, usability methods have been made available beyond the traditional confines of military, academia, and corporate R&D labs. If you’re interested in the history of user testing, check out Jeff Sauro’s usability timeline.
Why do you need to do user testing?
It’s a lot cheaper to catch potential problems early on in the product development life cycle. Tech companies typically follow an agile workflow that looks something like this:
This graphic represents how projects are passed through teams across the company. Product managers start by defining product requirements, which are passed along to the UX team. After finalizing design, the product moves through Development, QA, UAT, and Release Engineering. This cycle repeats itself in an iterative manner until the final product is complete.
Designers who rely on intuition and assumptions alone must wait until the release of each cycle to gauge feedback, potentially wasting a lot of resources. Reversing the damages of poor user experience is an expensive and arduous process. Once layout and patterns are established, designers must thoroughly capture evidence of poor usability and convince others to implement new patterns.
“If you think good design is expensive, you should look at the cost of bad design.”
— Ralf Speth
User testing was developed as a quick and inexpensive solution to mitigate some of these risks. You can validate assumptions, steer product and design direction more confidently, and uncover hidden issues using actual user feedback. With that said, user testing has limitations due to the artificial testing environment and small sample of users. It will never be a substitute for feedback from released products where large numbers of users are engaged with real data.
Get started with free tools
The simplest and most inexpensive method, paper prototyping, allows nondesigners to be involved in creating a simple interface. I prefer using wireframing and prototyping apps built specifically for digital products. We’ll start by using apps that are free or have free trials:
- Wireframe: Sketch, Balsamiq, or Draw.io
- Prototype: Invision
- Screen-sharing for remote testing: Google Hangouts, Skype, or Validately
Step 1: Identify reasons for testing
It’s important to first identify what questions you are trying to answer through user testing. Start by looking for potential pain points, complex user flows, and other ideas needing validation for your product.
With questions in mind, researchers typically start by choosing the best testing method and creating a usability test plan. Since we lack resources, we’ll stick with one of my favorite testing methods, thinking aloud. As Jakob Nielsen says in this article from the Nielsen Norman Group, “In a thinking aloud test, you ask test participants to use the system while continuously thinking out loud — that is, simply verbalizing their thoughts as they move through the user interface.”
Step 2: Create a script
A script is a printed document for moderating a user testing session. Your script should contain an introduction stating research goals and house rules, some probing questions, and then jump into performing key tasks. For example: “Hi Fiona! My name is Julie and I will be moderating this session today. Thank you for taking the time to participate in this session; your feedback is very valuable. Before we begin, I would like to go over a few items with you…”
Let’s pretend that I’m running a testing session for an app called DogeMaps. After identifying research goals in Step 1, I will brainstorm some questions around these themes:
- Background/Context: “How do you usually get around the city? Do you use any apps on your phone?”
- Expectations: “Before clicking on the ‘search’ button, what do you expect to happen?”
- Usability: “How do you save this location for future reference?”
- Rating Scale: “How difficult (1) or easy (5) was it to search for San Francisco on this map?”
Here’s a draft script to help spark some ideas. If your product is complex, prioritize your questions and conduct multiple rounds of testing. For a 20–30 minute session, aim for one or two intro questions and three to five tasks.
Tip: Pay attention to the wording and avoid leading questions that influence user behavior. For example, “How do you scroll to see more results?” suggests that I can scroll. A better question might be “Are there more results on this page? If so, how do you see these results?”
Step 3: Create wireframes and a prototype
Wireframes should focus less on visuals and more on layout, user flow, and copy. Keep your wireframes simple to save time iterating through prototypes. The general rule of thumb is to create high-fidelity mockups after validating core user flows.
The next step is to create an interactive prototype using Invision. Check out Invision’s QuickStart Guide to learn how to link wireframes together. The prototype must follow the same order of questions outlined in the testing script. If task #1 is “How do you log in to DogeMaps?” then the prototype’s first screen should be the login and the second screen should be the dashboard.
If task #2 is “How do you change your password?” then the dashboard needs to link to the password reset page. Imagine users trying to complete task #2 by clicking everything on the dashboard. It’s not necessary to link every button on each page; simply take notice and redirect them toward the right path.
When your prototype is complete, grab the private URL for the prototype and save it. In Invision, make sure to “Hotspots enabled” and “Prevent hotspot hinting” are both turned on.
Step 4: Recruit users
Brainstorm the target audience for your product. For instance, LinkedIn’s audience is Gen X and Gen Y professionals, while Snapchat’s audience is Gen Z teenagers. Provide incentives if you have a budget. Larger companies like Google or Airbnb typically incentivize users by offering a generous gift card. If you don’t have a budget, here are a few tips for recruiting.
If you already have users…
Identify people who are actively engaged with your product and send them a short message/email. Offer a sneak peek into the latest features and the opportunity to shape your product’s direction. Engaged users are more likely to give feedback if they have a vested interest in your product; your success equals their success. If users aren’t engaged, explain how your product brings value or solves their problem.
If you have no users…
Think about places where your target audience congregates either in person or online. Reach out to special interest clubs, friends of friends, Meetup, Reddit, online forums, etc. Here’s a sample message for recruiting:
“Hi Billy! I’m the founder of DogeMaps and I noticed that you were an avid user of GoogleMaps. I would love to get your feedback on DogeMaps — it’s an app that saves you 50% more time commuting than GoogleMaps. Would you have 20 minutes to hop on a Google Hangouts call? If so, what are the best available times for you?”
If all else fails, either broaden your user base, ask friends and family, or provide whatever small incentive you can. The logistics around recruiting users can be quite time consuming from my experience. If you have a larger budget and plan to recruit many users, companies like Usertesting or Ethnio can make your life easier.
How many users should I recruit?
Three to five users per round is enough to catch most problems. Spread your resources across different rounds of testing if you manage to recruit more users. Lastly, you may want to tread carefully through legal waters and have users sign a consent form.
Step 5: Set up your testing environment
It’s always preferable to conduct in-person testing sessions at a quiet office space. Cafes are popular, but can be distracting with all the background noise. Wherever you choose, make sure there aren’t too many distractions.
Remote sessions are easier to recruit for and can be conducted in the convenience of your own home. However, be prepared for technical glitches and always have a backup plan. I’ve experienced users unable to connect screen-sharing, mics not working, and prototypes malfunctioning.
Always conduct a trial run!
I’ve discovered prototypes that were incorrectly linked during actual user testing sessions. Don’t make the same mistake—always conduct trial runs to make final script and prototype revisions.
Step 6: Conduct the session
I usually send a reminder email the night before a remote testing session. It includes the prototype URL and instructions on how to set up Hangouts/Skype/Validately. The faster you get through logistics beforehand, the more time you have for asking questions.
Reminder emails are also useful to reduce no-shows, and I’ll typically wait 15 minutes before calling off a session. If it’s your first time moderating, you might feel nervous or unsure of what to expect. Don’t worry, you’ll progressively develop your moderating style and learn how to ask insightful questions.
A great way to develop user empathy across teams is by encouraging people outside of UX (project managers, developers, QA, etc.) to observe these sessions. Keep at least 15 minutes between sessions for users who run over their allocated time.
Find a notetaker
It’s impossible to take good notes and moderate at the same time. An ideal session should have at least three participants — one user, one moderator, and one notetaker. The notetaker’s job is to record what the user says and does, especially noting behavior that deviates from what the user says. For example, a user might say “I think the search button is very intuitive” after clicking seven different areas on the hunt for the search button. Be cautious of what users say and focus more on what they do.
If you can’t find a notetaker, use recording programs like Quicktime or Validately to review your sessions later. Be sure to disclose this both in your recruiting email and on your consent form.
Step 7: Distill key findings
The final step is to distill and share key findings using a Feedback Capture Grid. Draw a grid with the four quadrants and symbols above. The top left quadrant represents “user likes,” the top right represents “constructive criticism,” the bottom left represents “questions,” and the bottom right represents “ideas.” Transfer your notes onto post-its and place them into different quadrants, grouping together common themes of post-its. The end goal is to identify patterns and spark ideas within each category.
Revise, test, repeat
User testing will leave you with actionable next steps for your product. Products should start development when usability issues are addressed and general feedback is positive. If feedback is neutral/negative or the value proposition isn’t clear to the user, revise your prototype and run another round of testing.