Emulators and simulators vs. real-device mobile testing

by John Wargo

When testing mobile applications, there are two primary ways to experiment with the specific hardware and software conditions of various mobile devices and operating systems: emulators (or sometimes simulators) and real devices.

This isn't an either/or decision. You'll want to use both methods if you plan on having a mobile testing strategy that is both cost-effective and accurate. When and where you use them in your testing process will partially depend on your app's specific quality assurance needs and your budget.

However, the pros and cons of emulators and real devices are definitive, and so are the best practices for using them. Here are the advantages and disadvantages of each method along with the industry-wide best practices.

Emulators and simulators

Emulators and simulators are desktop applications that allow you to test mobile applications by imitating the behavior of real devices and operating systems. Emulators create full virtual machine versions of the mobile devices you want to test. These VMs try to duplicate the entire inner architecture of the device. Simulators, on the other hand, are lightweight tools that duplicate only certain features of a device without trying to replicate the architecture of the device.

Neither has the ability to accurately test sensor behavior or real-world hardware performance.

Pros

Emulators and simulators are cheap (basically free) and usually fast to set up and run tests on. They have many options for using different configurations, operating systems, and screen resolutions. This makes them the perfect tool for quick testing checks during a development workflow. You can delve into code, file structures, and databases in real time, allowing for nuanced inspection of whatever feature you're testing. The emulators or simulators are great for sanity-testing an app's user interface and catching the most common bugs.

Cons

Here are the key blind spots for emulators and simulators:

  • They can't emulate every sensor type.
  • The sensors that they do emulate are not going to provide representations as accurate as real devices.
  • They can't emulate the true CPU, GPU, and memory performance of a real device.
  • They can simulate network speeds, but not the many variables of real-world network connectivity such as the impact of network-related events (e.g., incoming calls, text messages, etc.) and different network technologies (e.g., HSPDA, WCDMA, UMTS, LTE).
  • Emulators are typically a "plain vanilla" version of the OS and often do not reflect the specific hardware and software features of each supported device.
  • There are some third-party libraries that require a specific type of hardware, so they can't be run on an emulator.

Emulators can also be slow at times because they have to imitate the hardware as well as run the software.

It's especially hard for Android emulators to be accurate when manufacturers are allowed to modify the Android UI for their own branding and design purposes. Because of these various inaccuracies that are possible when testing only on emulators, it's important that you do rigorous real-device testing. That's why mobile testing expert Daniel Knott suggests you flip and revise Mike Cohn's testing pyramid when building mobile apps.  

Real devices

Android Studio and Xcode both have the ability to let you connect to a device and run your app. While it's difficult to duplicate the exact conditions affecting every user's device, a real device is a much more accurate representation of how your app will run in the real world when compared to emulators.

The key challenge with real-device testing is figuring out how to do it methodically and efficiently. Buying dozens or hundreds of devices will get expensive, and testing them one at a time on a desktop computer isn't very efficient. As you scale up your mobile testing efforts, you'll need to consider a real-device cloud testing service or build your own test cloud. 

Pros

Testing on the actual devices your customers are going to use to run your app addresses all of the blind spots that were listed as cons in the emulator section. Real-device testing allows you to:

  • Accurately test the sensor functionality of your app if the device has all of the sensors that the app uses
  • Run realistic performance and user experience tests
  • Run realistic network connectivity and network interaction tests
  • Load any OS flavor or version that the device supports
  • Have fewer chances for false positives and false negatives

As you'd expect, real devices provide the most realistic user experience simulation for your tests. Duplicating some bugs on a real device may still be hard given the myriad of variables affecting mobile devices, but on an emulator, duplicating certain bugs is impossible.

Cons

Real-device testing can get real expensive real quick. Even if you ignore the likelihood that your user base will probably have hundreds of different devices (if you have an Android app), you would still need to buy new devices almost every month. Best practices call for testing on up to 40 devices in your market and replacing roughly 30% of those each quarter to stay up to date.

Then you need to maintain those devices and link them into a parallel testing farm if you decide to build your own mobile test lab. However, you can circumvent most of these disadvantages if you use a device cloud service.

Mobile device cloud testing services

There are very few downsides to using a mobile device cloud testing service. Any machine can connect to the Internet and use the device lab on demand (or according to subscription terms), 24/7. Given the heavy load that testers sometimes put on emulators and the large number of devices they need to quickly test in parallel, cloud-based emulator testing is a reasonable option as well.

If your team lacks the resources and skills to build, update, maintain, and secure an on-premises mobile device testing lab, then you're going to need to choose a device cloud service as you scale your mobile testing operations.

You can start with a handful of your users' most common devices (which is a lot easier if you're just testing iOS apps), but eventually you're going to need a larger device lab if you want to ensure quality experiences for all of your users.

Of course, all situations are fluid. If you don't have the budget to buy many devices, you may have to use emulators when, ideally, you'd be using real devices. Just be aware of the significant risk that poses to app quality.

Best practices

Overall, when should you use an emulator and when should you use a real device to test your mobile apps? Here are some general guidelines:

Use emulators for:

  • Very basic functional tests and test automation
  • Early user interface tests to verify the look and feel
  • Debugging flows as you're testing
  • Initial quality assurance

Use real devices for:

  • Testing changes to the architecture
  • The bulk of your manual testing
  • The majority of your automated testing efforts
  • Testing device-specific elements such as the hardware and software resources
  • Performance tests (network, load, stress, etc.)
  • Network performance tests
  • Anything else

Some tests you'll want to run on both emulators and real devices, so it's important to write many of your test scripts so they can run on real devices, as well as emulators and simulators.

Additional resources

There are several other resources that can help you further understand the details of testing with emulators, simulators, real devices, and device clouds. Be sure to read TechBeacon Learn's guide to building or buying a real-device testing lab in addition to exploring these resources.