A guide to crowdsourced testing and its benefits for mobile apps

by Justin Rohrman

Software companies sometimes look to outside help for cost savings in their mobile testing process. Perhaps they don’t have the expertise in-house, or maybe they need coverage outside of normal business hours. Or it could be that outsourcing is a good option because outsourced software testers often offer cheaper rates than in-house software testers. They can be briefed and scaled up faster than the normal hiring process.

A similar option to outsourcing is crowdsourcing, which might be a better solution if you need something cheaper than outsourcing that has the ability to scale up or down at a moment's notice.

Let's talk about how crowdsourced testing works along with its benefits and potential problems. In just a few minutes, you'll find out whether or not it's the right strategy for your mobile testing team.

The crowd

The crowd, essentially the freelance testers that a service employs, is the group of people who will be testing your product. Unlike with in-house or outsourced testing staff, you may not have direct control over which specific testers are in the crowd. However, you can often request that the testers have experience with certain platforms.

For example, you may only want testers who have experience using iPads with iOS 9 or testing e-commerce products. These search criteria help the service provider build a team with the necessary skillset.

"Private" crowd testing

Some companies try to form an internal crowdtesting group that isn't made up of freelancers, but rather some combination of their own technical staff, including programmers, testers, product managers, support team members, and even salespeople. When these team members are working together regularly for test runs, the company has what is often called a "private crowdtesting" group.

This is the DIY version of crowdtesting and might be the only crowdtesting option for companies that have heavy restrictions on making code available for testing outside their firewall. Setting up the processes and architecture for internal crowdtesting along with maintaining participation among team members who are more focused on their main job makes private crowdtesting a challenge.

However, if you can pull it off, the crowdtesting process of an in-house team is a lot faster to run when you don't have to brief the testers about the app as much as you would with an external team.

Crowdtesting platforms

Each crowdtesting service has to provide a platform for two types of users: the company buying the service, and the freelance testers. Typically, the customer and the testers will have an intermediary who manages the freelancers. Each testing session usually starts with an on-boarding meeting and ends with debrief meetings (which we will talk about shortly).

Customers will also get bug reports from the testers at the end of a session. In the on-boarding process, customer firms will need to provide documentation, known bugs, test cases, and any other information required for a smooth, clearly defined testing process.

While there are many general similarities among crowdtesting platforms, you'll also need to research each platform that you're considering in order to fully understand how they all operate. The best-known crowdsourced testing platforms are the following:

Crowdsource ceremony 

Every crowdsource testing provider should observe certain ceremonies:

  • On-boarding
  • Strategy
  • Debriefs

You probably want to steer clear of service providers that don't do so.

On-boarding is like a first date. You talk with the company offering the crowdtesting service and get to know the people. Software companies need to learn about the process the service provider is offering to see if it meshes with what they need. This is also a time for the service provider to learn about your needs to ensure that it can help you with your testing problems.

Strategy meetings will cover your release cycles and how to incorporate bug reports from the crowd into your development process.

Debriefs are similar to a retrospective in Scrum. Companies should talk with the crowdtesting service provider to find out what works, what isn't going well, and what specific things can be done differently to make the next cycle more useful.

Crowdsource billing

Crowdtesting services are often paid based on a cycle. For example, a company that needs five testers to work on new email integration features for a web app will get one rate. A different company that needs two testers to write Selenium Webdriver scripts during a development cycle for three cycles while it is hiring someone will get a different rate. Your rate will be negotiated with the crowdtesting service provider based on the difficulty of the work, how many testers are needed, and how long the engagement will run.

When to select crowdtesting

Now that the basic features of crowdtesting are clear, the next step is to do a little research about your team and the crowdtesting services. Knowing these key pieces of information will help your team determine whether a crowdtesting service is the way to go for your mobile testing needs.

Look at your usage factors

Usage factors are things that help you think about how you use software testers. Do they work mainly in an early or late stage of development? What is the complexity of your project? How long will the project run? What market do you work in? How much testing is happening in-house? What's your available budget?

Perform a cost analysis

A cost analysis will show approximate numbers for the cost of a full-time employee per year versus the cost of a testing contractor or a crowdtesting provider. Estimate how many release cycles you might need this tester for and approximately how many hours of work the tester might perform. A full-time software tester somewhere in the middle of the pack might cost $75,000/year plus expenses for benefits, tax, and anything else that comes up during the year. A contractor might run somewhere around $50 an hour, and a crowd might cost a few hundred dollars to a thousand per cycle.

Look at crowd service availability

Each crowdtesting service provider is dependent on the quality and availability of its crowd. Can it recruit enough people for your project quickly enough? Can it recruit people with the skillset you need? Can it recruit people that have access to the devices and platform you need? Make sure that any potential provider can satisfy these needs before buying. Also, think about how you might want to scale the project up or down to be sure your provider can handle this before you enter into a contract.

Once you've researched these key areas, you should have a pretty good idea of whether or not crowdtesting is an economical option. A company with a small budget that is very early in the development cycle, or a team that only needs people testing for a few release cycles, is a good candidate for crowdtesting. A company in the middle of a release cycle that is behind and feels as if it needs more testers on a complex project is not a good candidate for crowdtesting.

Successfully using the crowd

Each crowdtesting cycle should have a flow that includes plenty of time, love, and care so that the crowd is focused on your needs and values. There are three main aspects to the flow of crowdtesting, similar to the ceremonies discussed earlier. They include on-boarding, the test cycle, and a debrief after each test cycle. These steps need to be done in order and should not be skipped. 


Software development cycles have a find-fix-retest loop. Each time around this loop reduces the time for testers to find important bugs and the time developers have to improve code. The on-boarding process is designed to reduce the amount of time spent going back and forth fixing mistakes and misunderstandings about what needs to be done.

Testers working in the crowd will be information-hungry. You should provide their manager with accurate and up-to-date documentation about your product and the changes they need to be working on. You will also want to include information about where test environments or downloads can be found and what specific platforms need to be covered during the test cycle.

During on-boarding, you will also want to cover logistics—the number of testers you need, how many test cycles you are initially planning, what skillset the crowd needs to have, and what devices and specific platforms need test coverage. A little planning and design up front will save you a lot of pain later.

One last thing you'll need to think about is how you provide access to your app. If privacy restrictions prevent you from giving crowdtesters access to internal staging systems, they'll have to test the app in your production environment, and you'll have to build a beta version of your app that works in that environment. Even more extra work will be required if you want to create an isolated chunk of the production environment for the beta requests coming in during the test cycle. 

In a test cycle

A good test cycle will provide a rush of information about your product. Each bug report will be reviewed by a project manager at the crowdtesting service company to make sure that the format is usable and to reduce the possibility that you are getting duplicate bug reports. Within your own development team, someone, probably a tester or developer, will have to review each bug to make sure that it can be reproduced—and that it is a bug that you care about.

Although testers have incentive to find valuable bugs, you're not paying for 10x testers. They still may not find the most important bugs because the crowd lacks the advanced platform knowledge or exploratory testing expertise. Bug reports may also not be as detailed as you need them to be to reproduce the bugs. This can happen if there wasn't enough clarity in the on-boarding process, if testers are heavily motivated to value quantity of bug reports over quality, or if they just don't possess the experience to know what details to provide. 


In the debrief, you will meet with your crowdtesting project manager to talk about how the test cycle worked: Were the bug reports useful? Were testers working on the right parts of the project? Were the right testers used on the project? The crowdtesting service company may have some feedback for you as well related to your responsiveness to questions and your thoroughness in providing useful documentation and environment availability. Debriefs will end with specific tasks for both parties so that the next crowdtest cycle can be just as smooth as, or better than, the last.

On-boarding, managing the flow of a test cycle, and honest debriefs are the basics of managing crowdtest services. Other aspects of using the crowd, such as learning the crowd platform and refining the bug report format, will come over time. The important thing is to not skip these three steps.

Getting a successful crowdtesting process started takes a few steps:

  • Discover your testing usage factors
  • Perform a cost analysis
  • Study the crowd availability

After that, run the test cycles one by one with the three steps described earlier. 

Additional resources

Here are a few links to help you learn more about crowdsourced testing and how companies augment their development staff with a crowdtesting service.