A guide to automation testing frameworks—and how to build yours

by Bas Dijkstra

Anybody even remotely involved with test automation will have heard the term framework. But do you really know what a framework is, what benefits a well-thought-out framework can bring to your automation efforts, and the right way to go about creating one?  

Read on for answers to these questions and to clear up a few misconceptions you might have as to what a test automation framework is and what it can do for you as part of your software development activities.

So what is a framework, anyway?

There are many different definitions for framework out there, but here's the one that is the most widely accepted in the test automation space:

"A test automation framework is a collection of interacting components facilitating the creation and execution of automated tests and the reporting of the results thereof."

In other words, a test automation framework encompasses everything you need to be able to write, run, and analyze automated tests, except for the tests themselves. But before diving deeper into the components commonly found in a test automation framework, you need to answer another important question.

Why should you use a test automation framework?

Using a well-designed framework can bring significant benefits to your test automation efforts, in the following ways:

  • It minimizes the time needed to write and maintain tests. With a good test automation framework in place, creators of automated tests will no longer have to worry about things such as synchronization, error handling, reporting, and environment configuration, since the framework provides these for you.
  • It improves readability, reusability, and maintainability of the test automation code. By reusing the code in existing libraries and components for error handling, synchronization, and reporting, your test code will instantly be more readable and maintainable.
  • It provides an organization-wide guideline on how to write automated tests. I've worked with several organizations that did company-wide automation implementation yet did not see the improved efficiency they had hoped for, simply because they had a "to each their own" mentality. Having a test automation framework that's flexible enough to be adopted by teams working on different components (or even projects) can provide significant benefits with regards to knowledge and resource sharing.

In short, teams and organizations that want to be serious about their test automation will greatly benefit by taking the time up front to think through their framework design before diving deep into creating automated tests without the necessary scaffolding provided by a solid framework.

Common components of a good test automation framework

Now, on to the most common ingredients of a test automation framework. Note that not all of the components mentioned below must be present in every framework. Some frameworks might include all of them, and some will have only a couple of these components. Other frameworks will include components that are not in this list at all. Whether you are using an open-source framework or commercial tools that include some or all of these capabilities, these are the components that I see featured most often in test automation frameworks:

A unit testing library

Even if I'm not writing unit tests, unit testing libraries make up an essential part of almost any test automation framework I have created or worked with. They can be used to:

  • Define test methods, often through specific method annotations such as @Test (in Java) or [Test] (in .NET).
  • Perform assertions. Most unit testing libraries offer methods that will perform the actual assertions that determine the end result of your automated tests, so you'd do well to make use of these, instead of writing your own.
  • Run the tests. In most cases, unit testing libraries offer a test runner that makes running the tests you created very straightforward, whether you're running them from within your IDE, from the command line, or through a build tool or continuous integration (CI) system.

Examples of often used unit testing libraries are JUnit and TestNG for Java, NUnit and MSTest for .NET, and unittest (formerly PyUnit) for Python. Unit testing libraries are available for virtually every programming language.JUnit and TestNG for Java, NUnit and MSTest for .NET, and unittest (formerly PyUnit) for Python. Unit testing libraries are available for virtually every programming language.

Libraries for performing integration and end-to-end testing

When you're looking to create integration, API-level, or end-to-end automated tests—for example, those driven through the user interface of an application—it's often a good idea to make use of the features provided by existing libraries. These libraries make interacting with your application under test much easier by abstracting away all the coding needed to:

  1. Contact the application
  2. Send requests to it
  3. Receive the resulting responses

Some examples of well-known libraries in this category include: 

  • REST Assured and Karate DSL for Java-based API-level integration tests
  • Selenium (support for most major languages) and Protractor (specific to JavaScript)

Libraries supporting behavior-driven development (BDD)

These libraries are used to create executable specifications—i.e., they turn readable specifications of expected behavior (features and scenarios) into executable code. These are not test tools themselves, nor do they interact with your application under test. Instead, they are specifically used to support the BDD process and are often (mis?)used to create living documentation within the scope and intent of automated tests.

Well-known examples of BDD libraries include:

  • Cucumber (support for most major languages)
  • SpecFlow (for .NET)
  • Jasmine (for JavaScript)

Stubs, mocks, and virtual assets

During the creation of automated tests, if you find yourself...

  1. Wanting to isolate modules under test from connected components (a pattern that's often used in unit testing)
  2. Having to deal with critical yet hard-to-access dependencies (a very common phenomenon when writing integration or end-to-end tests for modern distributed applications)

...you might want—or even have to—look into creating mocks, stubs, or virtual assets that mimic the behavior of these connected components. Handling mocking and stubbing goes beyond the scope of this article, but it is important to know that mocking, stubbing, and service virtualization tools are increasingly a part of automated testing frameworks.

A solid test data approach

One of the most difficult problems when it comes to creating automated tests is how to handle the issue of test data. The higher you go in the test automation pyramid, the more difficult it becomes to ensure that the test data required to perform a given test is present or created when tests are run.


The test automation pyramid

There is no single approach to solve all test data problems, but having a solid approach with regards to test data management is imperative to the success of your test automation efforts.

This means that your test automation framework should offer the necessary means to create, look up, or inject test data for the tests that you'd like to execute. Note that the use of simulation can significantly simplify your test data challenges, since the mocks, stubs, and virtual assets you create are under your own control, whereas actual dependencies might not be so easily configured to suit your testing needs.

A reporting mechanism

No test automation framework is complete without suitable reporting in place. After all, what is the use of executing automated tests when there is no way to review and interpret the results?

There are many options available for reporting on the results of your automated tests. When you're tasked with deciding which reporting mechanism or library to incorporate into your framework, the most important question to ask is: "Who is the intended audience of the report that's being generated?" Different reporting mechanisms are targeted toward different consumers:

  • Reporting provided by unit testing frameworks such as JUnit and TestNG are primarily intended to be interpreted by other systems, e.g., CI servers (more on those later). These reports are often generated in XML format, which can easily be consumed by other software, but less so by humans.JUnit and TestNG are primarily intended to be interpreted by other systems, e.g., CI servers (more on those later). These reports are often generated in XML format, which can easily be consumed by other software, but less so by humans.
  • Many commercial tools offer built-in reporting capabilities to compliment the supported frameworks, such as UFT Pro (LeanFT), which provides human-readable reporting for JUnit, NUnit, and TestNG frameworks.JUnit, NUnit, and TestNG frameworks.
  • Reporting provided by third-party libraries such as ExtentReports or Allure can be used to create test result reports that can be interpreted by humans, including visual representation methods such as pie charts and screenshots.

Commonly used implementation patterns

Apart from the test automation framework components I've covered, various additional mechanisms make creating, using, and maintaining automated tests much easier, including:

  • Wrapper methods for reuse and centralized error handling. As an example, when using Selenium WebDriver, it makes sense to create custom wrappers around the Selenium API calls to handle timeouts, exception handling, and fault reporting. These wrapper methods can then be reused by the creators of the actual automated tests, so they do not have to concern themselves with these complicated matters and can instead fully focus on creating valuable tests.
  • Abstraction methods for improved readability and maintainability, and hiding low-level implementation details. A prime example of such an abstraction mechanism is using Page Objects when you're creating Selenium WebDriver tests. The goal of these Page Objects is to expose actions a user can take on a given web page (such as entering credentials or clicking a button on a login page) to higher-level test methods, abstracting away the need to locate specific elements on a page. Similar abstraction methods can be used for other types of tests and applications as well.

Build, source control, and CI components

Apart from the components mentioned above, there are three other key components that ought to connect to your test automation framework in some way if you're hoping to create a continuous testing pipeline:

  • A continuous integration platform. This platform is typically tasked with periodically building the software and running a number of tests against the new build in order to give developers and other stakeholders fast and regular feedback on application quality as new features are added and existing ones are updated (or even removed). Examples of popular CI platforms include Jenkins, Atlassian Bamboo, TeamCity, CircleCI, and Travis.
  • A source code management solution. Nearly all development teams have a source control/version control system. They are used to store and version source code and provide a way of safeguarding code and facilitating collaboration. Since test automation involves writing code as well, it makes sense to use source code management for automated tests as well, for the exact same reasons that these systems are used when working on production code. Some of the most popular source code management systems are Git, Subversion, Mercurial, and TFS.
  • Build tools and dependency managers. Dependency managers assist in gathering and managing the dependencies and libraries that software solutions rely on (this includes the libraries that automated tests rely on). Examples of these tools include Maven, Gradle, Ant, NPM, and NuGet. Build tools assist in actually building the software from source code and dependent libraries, as well as in running tests. Maven and Gradle are examples of tools that act as both dependency managers and build tools.

How to build a framework out of these components

Finally, let's take a look at how to go about building a framework out of the components mentioned above. Implementing a new test automation framework starts with determining what is needed. Some questions that need to be answered up front are:

  • What tests does the framework need to support?
  • What are the requirements with regards to reporting?
  • Who will be responsible for creating the tests using this framework? What's their level of expertise?

When creating new test automation solutions in cooperation with my clients, I always advise to start small, review early and often, and expand from there. Also, it might be tempting to start creating a large amount of automated tests right from the start. It's far more important, however, to have the framework in place as soon as possible, thereby tackling the problems in test automation that often turn out to be much harder than the creation of actual tests. This is especially important for creating a test data management approach, which is without a doubt one of the hardest problems in modern test automation.

Resources for further reading

A well-designed test automation framework will make your automation efforts more efficient, but in the end it's the tests that provide the real value. Make creating them as easy as possible by building the right framework (scaffolding) for them. If you want to read more about benefits of using test automation frameworks and learn about the most common frameworks and components, take a look at the following articles:

In the next TechBeacon Learn unit, I'll demonstrate how to create a working test automation framework using four components featured in this article: Selenium, TestNG, Maven, and Jenkins.