When to Test What: Validating Standard Features & Game-Changers

Written by Valle Hansen

As a user researcher, I’m always inclined to say, “Test everything, all the time!” when people ask, “What/when/how should we validate with users?” That’s my pie in the sky: the place where there’s all the time and all the budget in the world to get every last detail or spec just right for the good of the user, the product, and, ultimately, the business. But that’s not real life. Projects run on strict budgets and tight timelines, and there’s not always a lot of wiggle room.

So, in the absence of all the time and all the budget in the world, we have to prioritize, compromise, and make some assumptions.

First things first: Know your users. Define your audience, zero in on their major goals, and understand what data they need and how they need to see it. (Learn more about quick audience definition activities in my previous posts 4 Ways to Kick-Start Lean User Research for Agile Product Teams and Tips for Lean Audience Definition with Time-Crunched SMEs, and read about data discovery in Chris LaCava’s post Effective UX Design in an Agile World.) Once you’ve got a handle on your users, you can get started designing and building for them, and figuring out what needs validation now and what can slide by for a few cycles without validation. But how?

Let’s take a fake product example. We’ll call it Pacemate. Pacemate is a fake fitness application for serious cyclists that they can use to track cycling activities in real time and compare data points across their own cycling history as well as others’ history.

Let’s say you’ve got a pretty good understanding of your target end-users after a brief discovery process that homed in on what serious cyclists want in a tracking application. These enthusiasts are interested in stats around endurance, time, elevation, pace—basically anything to do with performance. They’re not focused on how many calories they burned or even what level their heart rate reached. First and foremost, they want to not only track their performance but also understand how certain variables affect their performance, such as time of day, time of year, what they ate for breakfast, and on and on.

After some back-and-forth with your product team, you’ve decided on a feature set for your first release (MVP) and a five-sprint goal for completion.

Now that you’ve got a proposed MVP feature set, you’re anxious to get to the finish line. But you want to do right by your users in addition to your budget. So what should you start building, what should you start designing, and what should you start testing?

1. When to Assume: Use UI Patterns & Validate Later

UI patterns are good solutions to common design problems. For example, a wizard is a useful solution for helping users get through a stepped process. Patterns are extremely useful in creating experiences for users that are comfortable and familiar, and will not force them to learn new ways of using an interface for no perceptible reason or benefit.

Several components of most personalized applications already have tried-and-true patterns that you can follow without much apprehension about alienating your end-user. Features like account creation and even profile creation come with pretty standard design guidelines. If your profile and account creation promise no surprises, stick with those patterns.

More specifically, in a market that has several other competitive applications similar to the one you’re creating, focus your validation time and budget on those features that will be the game-changer for your product. Sometimes this might just be a feature that’s already in the market but has been executed poorly everywhere. With Pacemate, features like auto-tracking and manual-input diaries are wheels that don’t need to be completely reinvented—but should be retested eventually.

To validate features that use a proven pattern or that already succeed in the marketplace, it’s OK to wait until after you launch the MVP. Consider using asynchronous automated testing services like UserVoice, or simple analytics tracking to ensure that your users like what they’re seeing and doing.

2. When to Compromise: Build Back End While Validating Front End

Some components of data-rich applications really should be validated up-front to make sure that users will experience the data in ways that are useful to them, but it’s probably OK to get started building in tandem with validating. These are components that need user input eventually, but will be easy and/or inexpensive to refactor if we get it wrong in the first go-round. It’s the difference between accuracy and precision. Precision is typically easy and cheap to fix with user feedback in the real world, but accuracy is a different story. If your solution isn’t accurate, it will be painful to course-correct down the road.

With the Pacemate product, the deep dive into historical data is one of those features that can be built in tandem with design validation. Assuming you’ve done your due diligence in a data discovery phase, get the known foundational components of the architecture set up, such as APIs and communication channels, while testing how users want to visualize the data, how they want to filter it, and what they might do with it next. Once you’ve validated with users, you can continue honing both the back-end architecture and the front-end design. You might even end up with great ideas for new features or feature enhancements.

3. When to Prioritize: Validate First, Build Later

What’s the game-changer in your product? What do you have that no one else in the marketplace has? Maybe it’s a graph database? Maybe it’s another link in the Internet of Things chain? Maybe you’re developing some artificial intelligence that can speak cat’s meow?

The game-changer is where you need to spend the most time validating before you build it. Not only is it the main thing you’re betting the farm on; it’s also most likely a relatively difficult implementation. If it hasn’t been done before, you don’t have a template you can start from to understand what your users want or to build it. You might think it’s the Next Big Thing, but your end-users might not, at least not in the iteration you’ve created in your mind.

Revisiting our fake product, what does Pacemate have that all the other cycling apps don’t have? And how easy or difficult is that game-changer to build and, potentially, refactor if it’s wrong?

Pacemate’s major player is likely to be its recommendation engine, where users can get real-time recommendations on what foods to eat before a race based on their personal historical performance, or specific races that would give them the best performance. This is a powerful tool but also one that’s easy to get wrong, and difficult to implement. Make sure you get it set up the way your users want before you start building. You’ll save a ton of time and money on refactoring.

Validating product ideas is an art as much as a science. Figure out what’s easy, what’s difficult, and what’s already been done, and go from there. But make sure your users are along for the ride.