How mobile testing differs from traditional testing

by Matthew David

A software team may be very comfortable with their workflow and testing processes, but it's still a challenge to shift their thinking if they find themselves building mobile apps instead of traditional, desktop-focused software. Smartphones are still significantly limited in hardware, network connectivity, and screen real estate compared to desktops. Therefore, testing strategies need to be different for mobile.

In what ways do they need to be different? There's a lot to learn before you can completely answer that question, but the first step is to answer these four smaller questions:

  • What traditional testing types can I keep using for mobile apps?
  • What new testing types do I need to use for mobile apps?
  • How do I begin testing mobile-specific hardware?
  • What mobile-specific testing tools do I need?

I'll provide answers to these four questions and give you an idea of the foundational tools and strategies you need to have in place to test mobile applications effectively.

What traditional testing types can I keep using for mobile apps?

Mobile development and testing patterns obviously have their roots in traditional development patterns and architectures, including N-Tier, API, and MVVM (Model-View-ViewModel). There are differences, but it's still software. Here are four testing types that are relatively similar between mobile and desktop software:

Unit testing

Mobile or desktop—it doesn't matter. Every application is built with code. It's just JavaScript, C#, C++, Java, Swift, or some other language. Unit testing is a technique for breaking down code into small groups and running tests against the code. Nothing really changes if the unit tests are for a desktop app versus a mobile app.

The true strength of unit tests comes with automation. Many unit tests can be run quickly by a unit testing framework every time a developer checks in his code for regressions (anything that might have broken while modifying code). You can find more tips on unit testing on this page.

Performance testing

Performance testing answers several key questions. For instance, you can see how fast your solution starts, how well your solution does when a large number of users are requesting services, and how well your solution performs over networks. Many of the fundamental performance concepts are the same for desktop and mobile applications. But unlike unit testing, you'll need some extra types of performance tests when developing a mobile app.  

Network testing differences

The first challenge is the network. Desktop performance is relatively easy. The PC is connected to Wi-Fi and is seldom moved. A mobile device is frequently moved. Daily activities force our phones to switch between different Wi-Fi signals (with Wi-Fi coming in many different signal speeds), different cell tower data signals (2G, Edge, 3G, 4G, LTE), and places with no data connection, or dead spots.

So a big difference for mobile is this: Performance tools need to test a broad range of network types to emulate a typical user.

Device hardware performance differences

Device hardware performance is at a premium for mobile devices. Consider a PC as a baseline. The average PC is connected to a dedicated power source or has a large battery. It also has large amounts of RAM/memory, a large hard drive (terabytes these days), and a large screen.

In contrast, a mobile device runs completely from a battery. CPU-intensive solutions will drain a battery fast. Most phones come with only a few gigabytes of RAM and a baseline of 16 to 32 GB of hard drive space. In many ways, testing solutions on a mobile device is similar to testing software for a PC built in the year 2000.

So you can't do all your mobile testing using tools that let you run virtual devices on a desktop machine. The performance testing results won't be realistic.

UI testing

Modern mobile development requires an MVVM development pattern to separate the layers of an app such as UI and code. With that said, the methods for MVVM began with traditional development over a decade ago, starting with Model-View-Mapping (MVM). To this end, the UI layer for many applications is placed into a separate layer. With the UI in a separate layer (often XML- or HTML-based), specialized tools can be used to analyze the presentation of different form factors and screen resolutions. This is the case for many desktop applications as well.

Diagram of tha MVVM (Model-View-ViewModel) architectural pattern

Mobile web UI

One challenge specific to web applications that are also used in mobile browsers (which include most web apps), is responsive design. Luckily, you don't have to build and test two different websites as you did several years ago. The safest way to make sure your responsive design works for all mobile screen sizes is to use a CSS framework. These are well-tested open-source tools that shouldn't allow major failures in sizing and spacing as long as you use them right.

However, you should always manually test your responsive websites with Safari or Chrome's developer tools to see that everything looks right on various device dimensions (which those browsers provide in their developer tools). This is especially important if you're doing internationalization. You might have several different word and paragraph sizes depending on what language is displaying, so you need to check that certain languages aren't messing up the page design.

Native app UI

Native applications use code in the app to manage the design layout. Both Apple and Google have design practices they recommend you follow when creating a solution. This way, your app will function as expected on any form factor. Apple's responsive design pattern is called auto layout, and Google has documentation on how to best implement responsive design in Android. Like the CSS frameworks, these built-in patterns for Android and iOS should make your app look right on the myriad of different mobile devices.

Two approaches to mobile usability testing

For testing mobile UIs there are two approaches you can use. The first is real-time manual testing, where a person reviews each screen on the running app while following a set script. The second is to choose a testing tool that will take screenshots of each screen as the test is being conducted. The second technique allows for greater accuracy. All you have to do is write the test and then review the screenshots after the test is complete. You can quickly validate if the test passed and quickly review the screenshots for design issues.

API testing

At the core of many mobile applications are third-party APIs (application programming interfaces). The APIs used for a website, a desktop app, or a mobile app are often the same. The focus for an API is to be the business logic and glue for moving data back and forth between an app, cloud services, and back-office systems. Splitting up the logic of an app into different parts like this makes it much easier to code the final solution.

APIs are a good place to start when building out your test environment. There is no user interface, and the analysis is driven by entering and receiving text.

Testing APIs can be automated and run much like unit tests. There's not much of a difference between testing API usage in desktop apps versus mobile apps, but the availability or cost of third-party API usage can be an obstacle, so look into service virtualization tools if that becomes an issue.

What new testing types do I need to use for mobile apps?

There are a lot of areas of mobile application testing that are different from traditional testing and require specialized tools and strategies. To begin with, there are three app types you might be dealing with:

  • Mobile web apps
  • Hybrid apps
  • Native apps

I'm not going to tell you which type you should build for your mobile app, since that would be outside the scope of this article. If you haven't already answered that question, read Smashing Magazine's exhaustive review of the mobile web vs. hybrid vs. native debate, which has been updated in 2017.

Mobile web vs. native vs. hybrid apps

Mobile web apps are built using HTML5/JavaScript and CSS. The only difference is that you need to consider mobile browsers in addition to desktop browsers when testing your application and determining the web standards that are supported. You can test using many of the tools that are already familiar to your web development teams.

In contrast, a native app is a solution that is compiled into a file that is then installed on a mobile device. The app is typically available through a public app store (such as Apple’s iTunes App Store or Google’s Android Play).

There are other ways to build APK and APP files for Android and iOS. The competing tools are often referred to as hybrid solutions (a solution that can be made with one set of code, such as JavaScript, and run on multiple operating systems through the browser or transpiling to native code). Modern hybrid technologies include Xamarin, Apache Cordova, and React Native. I'm only going to focus on testing apps built with Xcode or Android Studio, the primary native app IDEs for Android and iOS.

In the past, there have been other mobile operating systems (such as Windows Mobile and BlackBerry). Unless you have a specific need, don’t develop solutions for these platforms, since their global usage is now below 0.7%, according to KPCB. KPCB stats also indicate that Android dominates global mobile usage, with approximately 2.5 billion active Android phone users. Apple comes in second, with 1 billion active users.

Learning Android and iOS ecosystems

A native app is in many ways very similar to a traditional desktop application. There is a single executable (APK for Android, and APP for iOS) that contains all of the files to run the app. For a desktop app, you might need to build separate apps for Windows, macOS, and maybe Linux. For mobile, you also have to deal with a duopoly: Android and iOS. And as with the desktop operating systems, each mobile operating system has a preferred set of programming languages. Here's a simple diagram of the languages that enjoy first-class support on Android and iOS:

However, if you haven't been developing in these ecosystems already, even if your team is familiar with Java, as many enterprise teams are, you still have to learn a new ecosystem of tools for both Android and iOS (although the Java knowledge will give you a head start in the Android testing space).

Dealing with multiple screen sizes

While desktop apps run on laptop-sized screens or bigger external screens, Android and iOS developers have to worry about less screen real estate and many different screen sizes. Let’s take iOS. You have the iPhone or the iPad—smartphone and tablet-sized screens—but it doesn’t stop there. Within these two form factors there are many variations:

  • iPhone 7+ and 6+ screens are 1,080 x 1,920px
  • iPhone 7, 6, and 6S screens are 750 x 1,334px
  • iPhone 5, SE, 5S, 5C, and 6SE screens are 640 x 1,136px
  • iPhone 4 and 4S screens are 640 x 960px
  • iPad Air/Retina iPad first-, second-, third-, and fourth-generation screens are 1,536 x 2,048px
  • iPad Pro screens are 2,048 x 2,732px
  • iPad Mini second and third-generation screens are 1,536 x 2048px
  • iPad Mini first-generation screens are 768 x 1024px

This may seem like a lot, but the number of variations for iOS screen sizes is tiny compared to the vast array of Android devices in use today (nearly 19,000). But for smartphones, at least Android has gotten a tight handle on the allowed sizes from various manufacturers:

  • Extra-large screens are at least 960dp x 720dp
  • Large screens are at least 640dp x 480dp
  • Normal screens are at least 470dp x 320dp
  • Small screens are at least 426dp x 320dp

You can find more information about Android screen sizes here.

Both platforms have built-in methods for dealing with this issue.  

iOS: When you are developing an app in Xcode, you can use Apple's Autolayout tool to assist with these different screen sizes. 

Android: Google provides similar built-in features to help manage the presentation of Android apps on different screen sizes. 

Leveraging these tools does not mean you don't need to test the layout of your app. Without real-device testing, you never truly know how your app will look on the user's device.

Testing mobile-specific hardware

When building a web app, you don't really have to think about the hardware. You just test to see if it works on various browsers and is responsive, with a great layout for multiple screen sizes. But building native apps means you have to account for mobile devices that come with cameras, GPS, gyroscopes, and many other sensors that desktops don't have.

Emulators and simulators

Since these hardware-level concerns need to be tested, the immense mobile device proliferation and fragmentation outlined in the last section poses a serious problem: How do you test so many real devices? You don't always have to.

Android Studio and Xcode both come with software that simulates or emulates real devices. These are great tools for rapidly seeing how your software will run and display for different phones and tablets.

Real-device testing

However, emulators won't give you a real understanding of how your app will run on a particular set of hardware. For this reason, the leading IDEs also have the ability to test apps on connected hardware.

For example, you can connect a handful of the latest Android or iOS phones to your computer and load a test version of your app onto the phone. The result is that you will see how your new app will run on the phone’s hardware versus the hardware of your desktop computer running the IDE. Your desktop will always be much faster—making some testing inaccurate for real-world usage.

And you also need to think about the significant differences in computing power between cheaper or older devices versus the latest premium devices. Testing your app on a real version of each is the only way to know for sure that the experience is relatively consistent.

Testing on real devices also opens up the opportunity to test the many sensors and tools found in a tablet or smartphone. Typical sensors and hardware to test for include:

  • GPS
  • Magnetometer
  • Gyroscope
  • Accelerometer
  • Proximity
  • Ambient light
  • Fingerprint sensor
  • Microphone
  • Cameras
  • Barometer

The list above simply cannot be accurately tested with virtual software. You must test on real hardware. 

Scaling real-device testing

The challenges with testing on real devices are scale and budget. It wouldn't be very efficient to load an app onto each different device you're checking whenever you need to run your test suites again. It also gets expensive, because you constantly need to buy new devices that arrive in the marketplace. Take Samsung, for instance; on average, Samsung releases a new Android phone every week! 

Luckily, there are tools and services that can help you solve these scaling issues with real-device testing.

Tools to help with mobile testing

If you don't want to try buying all the devices for your own mobile testing device lab and integrating that system seamlessly into your testing and development workflow, you can rent that type of service on-demand. Here is a list of cloud-based mobile device lab vendors:

Having some form of real-device testing strategy will be crucial when going from traditional testing to testing mobile apps. The biggest challenge of mobile testing is that you're working with a device-fragmented ecosystem with lower computing power and screen real estate, so you need to have tools in your testing arsenal that account for that. There's a greater amount of manual, real-device testing required in mobile testing, so you'll need to do some significant research into methods for easing that new burden.

Further reading

TechBeacon has some great resources for learning more about mobile testing best practices. Have a look at some of these articles if you need more information in some of the areas discussed above: