Mobile UX testing strategies beyond the phone

by Paul McAleer

Usability testing is a cornerstone of designing great experiences. It helps user experience professionals determine what functionality truly works and doesn’t work for target audiences. Because the practice of usability testing has been maturing for decades, many of its core principles and applications work in a variety of ways and on a variety of devices.

But there’s one little thing—okay, a big thing—that changed: our main computing devices. Instead of sitting in front of large monitors for a number of hours each day, the one-two punch of mobile phones and the wireless Internet made tech more portable and invisible. Thus, some practices and philosophies around usability testing for desktop web and desktop apps have become a little less useful. This multi-interface world demands a fresh look at the way we conduct usability tests in general, and what we’re looking to get out of them specifically.

This unit outlines one approach to building a mobile UX testing strategy with a focus on device handoffs. See which components from this approach work best for you, and go from there.

Setting usability goals and milestones

When usability testing becomes a part of your project plan, it’s tempting to start with the pure repeatability of it.

“Six people at a minimum? Check!”

“Script that covers common tasks? Check!”

“Figuring out time slots and availability? Check!”

And about 18 other checks!

But before you do that, take a step back and examine the overall strategy of your testing. And yes, you need a usability testing strategy.

Like other higher-level techniques, you’ve got to first define your overall goal and milestones to support that goal. This activity also ties into your overall design goal, so that’s worth bringing in. Three milestones aligned to your top ////top what?//// is a good number to start with (including metrics attached to each milestone if applicable). Here’s an example:

  • Design goal: Create an app that allows people in-store to quickly see maps, access coupons, and locate the items they need to have a fun shopping experience.
  • Usability testing goal: Ensure that our app is usable and useful within our retail environment.
  • Milestone 1: Evaluate the effectiveness of the home screen design with poor connectivity, as is the case in some stores.
  • Milestone 2: Ensure that maps load quickly and show users where they are within the store layout.
  • Milestone 3: Test search functionality to ensure that it understands customers’ terms and gets them routed to what they need easily.

As with anything digital, nothing is permanent. These milestones and even the broader goals might change as you continue to work through research and design. But it’s important to bring clarity to your work early on, and this type of framework is a great aid in that.

Research informs everything

Up-front qualitative research—which includes audience definition, behavioral analysis, and stakeholder interviews—is extremely valuable. It’s essential to understanding the overall problem, and it pairs nicely with quantitative research.

Your research keeps your usability testing strategy-focused. For example, when you first get a project involving nonscreen/voice interactions, it can be extraordinarily tempting to work it into every behavioral persona you’ve got:

“This one? Oh yeah, I can see them talking with our app to get directions … what? They already know where our stores are? Still!

Some good advice here is to let the research determine what you need to test, just as it informs your information architecture, wireframes, visual design, and everything else. If you’re including nonscreen interactions that aren’t realistic, you’re potentially wasting your time and focus.

This isn’t to say that you can’t explore the space, but if your audience isn’t using a home speaker or a smart device other than a phone, there may be other research tools to lean on. For example, you may have the opportunity to ask people in your usability testing wrap-up questions—simple, broad questions—about these devices and their interests. These could inform future research. The bottom line? Don’t get distracted by shiny objects! Let the research be your guide, as always.

Tactical tips and tricks

There is an art and science to usability testing. Single-platform usability testing can be relatively straightforward: There are one or two stimuli on a single screen. When you’re looking to shift the experience from one screen to another, or from a screen to a voice prompt and back, there are additional tactical elements to consider. Here are a few.

What are the most natural handoffs between devices/interactions?

Handoffs—starting a task on one device and switching to another, and possibly back again—may be common for your target audience. The shopping example we mentioned earlier requires these handoffs: Once shoppers find the thing they’re looking for in the store, they’ll interact with the actual product and (hopefully!) complete the purchase. Even in this instance, the entirety of the experience doesn’t exist on one platform, so those handoffs should happen at organic moments.

Is the underlying experience aware of the handoff?

This is a rich area: cohesively designed experiences that are aware of and might even anticipate these handoffs, allowing people to pick up where they left off even if time has passed. If users start shopping on a phone and add something to the cart with a smart speaker, they may expect that item to still be in the cart when they review it on their laptop. That’s a consistent concept that flows between devices.

The important delineation here is aligning tasks with mental models and design goals. If your testing strategy or plan can’t support the handoffs, you may need to mock them out or talk users through the feedback they’ll get from another device.

Many experiences across screens and interfaces aren’t this sophisticated, sadly. But that’s all right: If you know this is going in, you can include it as a part of your test. You should expect a little more friction, though.

Can you test with the actual devices/screens/interfaces in your sessions?

One more point that is completely logistical but still important: If you can test with the actual devices you’re designing for, that’s ideal. But there may be circumstances out of your control. Emulators and simulators can give you some really good data on the structure of the menu interface, but it wouldn’t give you any information on the physical control.

When you encounter roadblocks related to real-device availability, you’ll have to decide which parts of the experience have more testing value than others based on your goals. For instance, if a home speaker responds to an interaction on a watch and you can’t bring both into the test, you’ll need to analyze and prioritize what matters most in that experience. Is it the watch interaction? Is it feedback from the speaker? Is it what happens next?

What are the contextual clues a device can give you?

The biggest difference on mobile is the user interface: Touch targets must be larger, text generally needs to be bigger, and scrolling is common. Beyond that, there’s a philosophical differentiation. This is a tiny computer that knows where it is, what angle it’s being held at, the temperature, the current elevation, what devices are nearby, and more. So when considering tasks, you must take these things into account.

Notifications, for example, are a common part of the experience. So too are software update prompts, random alert boxes, and poor connectivity. We need to anticipate these contributing factors in our work even though they’re not ideal, because our users won’t be using our designs in an ideal world.

Given these challenges, you can see why putting a bit more thought into your usability testing strategy can be beneficial.

Additional resources

Usability testing is flexible enough to shift and change over time, just like the devices and interfaces we use. It’s important for designers and researchers to not just look at what’s been done before; instead, a strategy that’s based on the reality of our audiences’ needs can yield more fruitful usability testing results.

Here are some additional resources on mobile UX: