Which tests should you automate? Here's how to decide

by Feroz Luis

If you want to calculate the return on investment for creating an automated test, follow this simple formula:

Compare the initial and ongoing costs of developing, executing, and maintaining a set of automated test cases against the initial and ongoing costs of running the tests manually.

When calculating your initial cost, be sure to include both test development and licensing costs for any tools used in the process. Calculate your ongoing costs by measuring the work hours needed for the execution and periodic maintenance of the tests. Parallel test execution can compound your time-savings.

This unit contains specific advice about the value of automating nonfunctional tests, system integration tests, graphical user interface (GUI) tests, back-end tests, and others. But the core formula above should serve as a general guide as to whether a specific test in the context of your process, toolchain, and organization should be automated. 

Nonfunctional tests

These tests validate whether the application can withstand load and meet performance requirements. Some typical client-side and server-side values to measure include:

  • Memory usage
  • Bytes in/bytes out
  • Heap-size
  • Thread pooling
  • CDN usage
  • Database query performance

Network performance tests are also typically considered part of the nonfunctional testing suite.

Nonfunctional tests are typically automated, and carried out during the initial development cycles of the application. Once an application achieves performance stability and enters a maintenance lifecycle, some organizations will conduct these tests only when needed to save time. However, this can lead to performance degradation over time, so its important to constantly run and maintain any non-functional tests that are continuously needed to prevent performance degradation.

GUI tests

GUI testing checks the presence of various interface elements to confirm that they meet the design specifications. It does not test the functionality of these elements, but only verifies that they are present and in the right place.

GUI-specific test cases are not ideal candidates for automation. After the initial round of GUI test execution and debugging, further testing rarely reveals defects if there are no significant changes to the user interface (UI). So automating GUI tests has a low return on investment. Also, the validation of UI elements using automated tools is usually complex, time-consuming, and highly error-prone.

You're better off using tools to test the code behind the UI, rather than the visual representations themselves. The manual validation of UI elements (with a human eye) on various screen sizes is much easier and more accurate. But there are always exceptions that might make GUI test automation viable.

Middleware, API, and back-end tests

You use these tests to validate the underlying infrastructure of any application. Although these are good candidates for automation, be extra careful to identify areas that might be affected by newer code changes.

For instance, repeatedly testing API functions that query database tables and fetch values isn't very beneficial. But testing a stored procedure that can change due to a change in a table value is a better candidate for automation because you'll want to keep executing that test in subsequent releases.

Functional tests

Functional tests are written to validate the core functions of the application. The main types of functional tests include:

Unit tests

These tests validate a portion of code against specifications. Such tests are generally written by the developers to determine whether each function performs according to the desired behavior. Unit tests are almost always automated within the application development stack using unit testing frameworks such as JUnit, TestNG, RSpec, or Mocha.

System integration tests (SITs)

This includes all tests that validate functionality against functional specifications and verify the integrated system. Not all system integration and acceptance tests are candidates for automation; that varies depending on whether you can get enough value from executing certain tests repeatedly. Here are a few types of SITs that are usually executed frequently enough to merit automation:

  • Smoke and sanity tests: These are executed during every code release to ensure that the app's basic functionality remains intact and that the application is ready to be tested more thoroughly.
  • Regression tests: These tests are required to check that new code has not broken existing features. These tests cover most of the core functional flows.
  • Acceptance tests and end-to-end tests: Acceptance tests check the functionality of the app from the end-user perspective. They take the instructions of the user stories created in the app's design phase and use those to test the end-to-end flow of information. (The code to automate them is also written in the design phase in most cases.)

One steadfast rule to know when you should not automate a test is to look at whether the functionality you want to test is changing rapidly. If so, your test automation scripts will be brittle, since you'll be constantly changing how the code should be tested. If you are constantly updating some test automation scripts for a specific part of your codebase, you might want to review how much time those updates cost. If there's no end in sight for those updates, you might as well test those components manually.

Additional resources

The guidelines above can help you to identify which tests to automate and which would be too costly. But every project is unique in its challenges, so you should always carry out a retrospective to evaluate the ROI of all automation projects.

Consider researching this topic further to get an even better idea about which tests to automate, as the answer is not always predictable and can be context-dependent. Here are a few additional sources you can use to help you decide which tests to automate: