Deciding factors: Build or rent a real-device mobile testing lab?

by Justin Rohrman

Software developers are familiar with the build-or-buy dilemma. Teams that build end up with a custom-fit product that meets their exact needs and can even be extended in the future. The other choice, buying a solution, allows the team to jump into solving the problem instead of building infrastructure, scaffolding, and support to make it possible to solve the problem tomorrow. A manager makes a few Google searches, picks the product that fits her team’s needs at the right price, and clicks buy.

Mobile test teams have this exact problem in their testing device lab. Should they build a lab? Should they partner with a company that already has a built-out lab? This article will walk through the steps to build and buy a device lab, and discuss why you might to do a little of both.

Start with the customers and their devices

The first step in deciding whether to build a mobile-device testing lab or to buy one through a service provider is to assess the project's mobile hardware needs. Start with what customers will be using. If you have apps that are already live, then you can get this information with a handful of open-source user analytics tools, including:

Do your customers have one brand of hardware, a handful of options, or are they using whatever consumer products are available. Are they using standard screen sizes? What version of the hardware will most customers be using, based on Apple and Google’s customer research? Manufacturer, screen size, and operating system are all part of that decision.


Companies that support multiple hardware platforms, such as Samsung, HTC, and Nokia, will have tougher choices to make. Partnering with an external lab makes more and more sense as the variety of devices increases. In most cases, mobile test lab providers can easily provide a wide array of handsets and tablets without extra expense to the test team.

With Apple, this is less of a hassle. The first iPhone was released in June of 2007. Since then, there have been only 15 different versions of this product sold. Apple users tend to update their hardware within the last version or two of a product release, so test groups can still get away with a minimal amount of hardware in-house. Other manufacturers have more complex product lines, combined with customer bases that update less frequently. So there may be 5 or 6 different versions of a particular phone or tablet being used in production.

Companies that only develop software for Apple hardware will typically build their own mobile-device test lab. Apple's relatively simplistic product lines and control of both hardware and software make building a lab easier, because Apple sells the devices directly, along with support packages. Companies that want to control costs can take shortcuts with Apple—for example, buying an iPod Touch instead of an iPhone, then testing with wireless service only, with no need for a mobile phone contract. This is a trade-off that involves a little risk, but it does make building a lab much cheaper.

Screen size

After hardware platforms, start investigating the screen sizes that will be used in production. Web apps viewed through a mobile browser will have to be built with responsive design, which uses JavaScript, CSS, or entire frameworks to update the UI’s structure and usability based on the size and shape of the display. Native apps have tools in the SDK that will help your app adapt to a handful of screen sizes. Even Android has locked down the number screen size profiles to just four.

Companies that only need to support a handful of screen sizes, maybe two or three, can easily buy those devices and manage them in-house. Other companies that have to support any of the hundreds of screen sizes have two options. They can take a sampling of the screen sizes used by customers in production and buy the most common devices for their test teams. This allows teams to have real devices, but with a downside of missing occasional bugs related to specific screen sizes that they don't have access to. Alternatively, they can buy a mobile-device lab service that will give access to nearly any screen size that a customer would care about.

Operating systems

The Android market is splintered between device manufacturers and its user base. Although it is less common than it used to be, some hardware manufacturers still create custom versions of the operating system, including vendor-ware or custom browsers. Android OS fans buy unlocked devices from third-party carriers and customize the OS themselves or install customized versions created by the developer community. The vastness of the Android ecosystem means software developers and testers sometimes need a wide variety of devices, especially when writing applications with low switching costs, where insisting on a small list of supported devices is impractical.

The iOS ecosystem is much easier to cope with. Apple has years' worth of iOS releases and updates, but its customers tend to quickly install the latest update. A company working only with iOS might be able to get away with having two versions of each device they support: one with the latest version of iOS, and one with the previous version.

Businesses supporting massive numbers of users on multiple operating systems, including iOS and Android, are all but forced into buying a mobile-device service since there are so many devices to maintain. This usually means shopping for an affordable mobile-device cloud service.

Choosing the cloud

One key upside of a mobile-device cloud service is that it is nearly maintenance-free. Device cloud vendors take care of expanding their device lab to keep up with the consumer market, and they also maintain different device versions to keep up with software and platform development.

Before buying a cloud service, teams should make sure that the service:

  • Has customizable services that include as many devices as the team needs
  • Is priced based on usage and need
  • Supports easy access to real devices, emulators, or simulators
  • Provides the ability to use devices in parallel for test automation
  • Includes performance monitoring
  • Exhibits low latency when testing on cloud devices or emulators—comparable to local device or emulator testing
  • Provides exclusive access to test devices while the team is using the service, or provides a private test cloud option
  • Removes the team's app from a cloud device before another customer uses it
  • Can connect to the team's continuous integration toolchain
  • Generates testing reports

Even with a device cloud, it's not feasible to test on every single device, but testing the 10% of devices that represent 90% of the market is acceptable for most organizations.

Companies that want to buy their mobile test lab in the cloud will want to investigate these service providers:

The downside of cloud vendors is that availability of devices is constrained by your local Internet connection. Network problems and lapses in Internet connectivity will halt any mobile testing. Security and privacy issues can be difficult to navigate for companies in finance or healthcare. And, rather than a one-time fee, cloud users will pay monthly based on the resources they consumed.

Choosing real devices

A real-device lab creates the opportunity for deeper testing. Testers interact with cloud devices the way they would with any web app: clicking with a mouse, and typing with a keyboard. Testers holding a real device can tap, type, rotate, gesture, and walk around with the device just as a user will.

This device lab can be set up as a simple rack of charging stations with a check-in and protocol. This creates an environment where devices are accessible to the testing team as needed, and they are all accounted for at any point in time. A more sophisticated option is building a private mobile-device cloud. Private clouds are secure and available to testers at any time, but they require staff and expertise to build and maintain. Companies considering building a local mobile-device cloud should balance that implementation cost with the cost of buying a mobile-device cloud service. 

Making the decision

Here is the step-by-step for a build/buy decision:

  1. Take a look at which devices customers will use. This number can cover a variety of devices and appear intimidating. But when the usage statistics are analyzed, a few devices will have a majority of the market share.
  2. Investigate how many members of the test and development team will need mobile testing devices and how long they might need them. Larger teams with slow development cycles might be successful with a small number of devices. Smaller teams that have fast delivery cycles or support a variety of platforms and devices may need either a large in-house test lab, or a mobile-device lab service provider.
  3. Investigate, ideally on a trial basis, the actual experience of working with a partner. Most external labs are cloud-based offerings, meaning the device is plugged into a facility, then viewed on a laptop. The experience of using these emulators, simulators, and "hosted devices" can be very different from actually holding the device in your hands. Propagation delays, bandwidth, and look-and-feel issues can get lost when the device is accessed through a web page or on a laptop. Organizations with a large need may come up with a mix of lab and physical devices.
  4. Compare the immediate cost and expertise requirements for building a lab (including new devices as they are released) with the cost of working with a partner over a three-year time period, including both immediate cost outlays and total cost. Adjust for the quality of the experience and the amount of coverage available.
  5. Determine the best strategy and implement, either purchasing devices, signing a trial agreement, or both.

Keep reading: Building your own lab

Even if you decide to use a mobile-device cloud service for the bulk of your testing efforts, most organizations will still build some form of local device testing lab, just so they can quickly confirm certain features with their own hands and eyes. There is no one right way to build a mobile-device lab. But there are a few guidelines: Have a plan for device management that includes power, a check-in/check-out procedure, maintenance, a variety of devices that reflect what customers will use, and stay within a budget.

Here are a few case studies to consider when building your own mobile-device lab: