How to build a mobile beta-testing strategy

by Justin Rohrman

Today most of us agree on a few simple principles. We want to build small pieces of software, one at a time; release as often as possible; and get real feedback from customers. Some companies take that a bit further, trying to automate the process of integration, deployment, rollback, and monitoring, which is related to DevOps. Others want to release in a big bang or want feedback directly from people rather than software systems.

Deciding which is best for your organization and how to build a mobile beta strategy around that is the focus of many mobile app development teams today.

Getting started: Choosing the right feedback mechanisms

A strange as it sounds, the beta release is not about the release. Instead, it's about getting data back from customers. Without this, you could skip the beta, or just continue to work on the current plan and ship on time. Step one of any beta-release strategy is figuring out what feedback management needs to make product decisions and how to collect it.

Your data-collection strategy should have two main aspects: collecting data from users in your product, and talking directly with users.

Reporting plugins

Select a plugin such as Usersnap or BugMuncher so people can report problems from within your product. When a user discovers a problem, these reporting tools allow them to send a description of what happened, possibly with a screenshot and some information about their session, the browser they were using, and where they were located.

Production monitoring tools such as Nagios or Zenoss are typically involved here too so that the development team can track information about resource usage (memory, CPU cycles) and relate that back to a user report.

Customer communication

The “talking with people” option involves gathering customer information during a beta release. This means members of the development and product teams need to meet with customers when possible. Doing so gives the customer an opportunity to not only demonstrate the problems they encounter, but also to show how they really work.

Some combination of these two methods is ideal when possible.

Collecting feedback from beta users will give development teams a pool of data about their product. The next step is to build a strategy for sorting through the data to figure out the bits and pieces that are useful.

Beta-test management tools from Apple and Google

After creating a data-collection strategy, you’ll want to select tooling to help with beta-release management. These tools allow companies to give a selection of people access to products that are not available through iTunes or the Google Play store.

Native applications need a way to be downloaded. Ideally, this method ties into a reporting system. Apple's TestFlight software takes a collection of email addresses (up to 2,000) and provides a special link to download the latest build from the Apple Store. This gives the software publisher two streams to deploy to: the production build, and the invite-only beta build. It allows publishers to have a "beta list" of users and notify them continually about new features.

If you are on an Android platform, tools such as HockeyApp, Appaloosa, and TestFairy will help with beta-release management.

Collecting and refining bug reports

Once you have a way to get your mobile beta builds to customers and have started collecting data, you will need a way to make that data useful. The next step is building processes and a team to handle your new mobile-beta-release reports.

Most of the time, beta users will not be software professionals. The feedback from a beta trial might just be: "submit button broken," with no information about which submit button doesn't work or what the beta users were trying to do when they saw the problem. Some bug reports might be highly detailed, with very specific steps outlining the problem, but ultimately leading nowhere. And others will be repeats of bugs that have already been reported several times.

You will need to dedicate some time from your development group to sort through these issues. That group should include testers or developers, if not both. These people will spend some time each day reviewing the bugs that were reported from beta users through in-app plugins such as Zenoss, via email, or by talking directly with mobile users. Testers and developers will look for problems that are not reproducible, problems that are duplicates of other reports, and bug reports that might need more information from the user. Once the chaff is removed, the technical team will prioritize these bug fixes and plan for where they fit into the next release cycle.

Selecting appropriate beta partners

Creating a partnership with select customers will greatly improve the quality of feedback you get in mobile beta releases.

Beta releases are a tricky time. These builds are very close to production-ready, but not quite there. They will have bugs, incomplete features, and missing features. It takes a special type of customer to deal with these conditions willingly, and even more to work as a partner with the software maker to help improve the product and get it ready for production.

Startups should form beta partnerships with their first customers, whether they’re early adopters from the consumer market or a company. With early customers, it’s a good practice to involve them in beta tests, because the success of both parties depends on the software meeting expectations. The “tests” with these beta partners should include regular, in-person feedback; reviews of bugs; and status updates.

Larger software vendors with many customers can build similar partnerships with select customers. These companies will have to consider how much staff is available to visit these partner companies and how they will handle large amounts of feedback. Creating a third-level support group to handle reports from beta customers or having a special bug triage team will help make the work manageable. Third-level support teams are small groups of people dedicated to working directly with customers and making product changes quickly when needed.

If your customers are from the general consumer market, beta releases should be targeted at people who want early access to the bleeding-edge features in exchange for experiencing and reporting any bugs before a general availability release. If you want to get feedback from specific regions or users of specific device types, you can also learn a lot by doing that kind of segmentation.

Selecting the right customers and creating the right size beta program is a step toward success. If you need more guidance on building the right size beta program, read "How to rightsize your mobile app beta program."

Beta build management

Next, your development team will need to prepare a product build that is specially made for your beta customers.

The beta build you choose to release can be just another build, or something special just for beta customers.

To maximize the amount and variety of information you get, you will want to do some amount of tooling. This tooling will help you get better log information from mobile devices when something goes wrong, allow users to report bugs from within your app, and target who has access to beta builds. Here are some of the things you might want to include:

  • Improved application logging that displays things such as a time stamp, user that performed an action, user region, and platform information such as browser and OS
  • Plugins that allow easy bug reporting, such as Usersnap or Google Feedback

Some companies release two versions of a product at the same time to run experiments. Often called A/B testing, this allows development teams to get feedback on new ideas from a smaller set of customers before fully committing to a large change.

Beginning and ending beta releases

Next, you need to decide how long your beta trial will run.

Large games and apps will sometimes have a signup phase for people who are interested in participating in beta testing. Once the beta release is ready, these people get a link to download the special product version through beta-release tools such as HockeyApp and Test Flight.

Customers working with commercial or enterprise products will probably begin using a beta version of the product on some negotiated date. Web products that are accessed through a mobile browser can be set up so that the user doesn’t need to do anything other than navigate to a special URL. Native apps that are installed directly on a device will need to be managed with software such as TestFlight.

Beta-release trials can be ended by sending an email announcement and providing a new product installer, or simply by shutting off the beta communication channels of bug reporting tools and forums.

Additional resources

To recap, a beta-testing strategy generally includes:

  • Choosing who you want to get your feedback from, and how many
  • Building a process to collect and evaluate feedback
  • Designing procedures for initializing and ending a beta-test cycle

After that, things look a lot like a normal software release cycle. Take that first step and analyze the kind of feedback you need and how much you need. Things will fall into place after that.

Here are a few resources to read while building your beta-testing program: