What is Parallel Testing, and How to Implement It?

When it comes to software quality assurance, organizations lay a heavy emphasis on ensuring flawlessness in their products and reducing defects as much as possible. This is because many software organizations are leaning towards Continuous Integration/Continuous Deployment (CI/CD).  This is where automation testing proves to be beneficial as it reduces subsequent test efforts. Not only does it reduce test time, but it also increases test coverage. 

Parallel testing is an automated testing technique where you can test multiple versions of subcomponents of an application with the same input on different systems. The prime benefit of it is that it helps reduce test execution time and speeds up the testing process.

In this blog, we will discuss the following aspects of Parallel test execution:

• Parallel Testing definition
• Benefits of Parallel Testing
• Challenges posed by Parallel Testing
• Implementation of Cypress Parallel Test Execution

What is Parallel Testing in Software Testing?

Parallel testing or parallel test execution is a software testing approach in which multiple tests are run at the same time simultaneously. The goal of parallel testing is to reduce the time it takes to run a suite of tests by dividing the tests into smaller groups and running them in parallel. This can be achieved by executing tests on multiple physical or virtual machines or by dividing the tests into smaller chunks and running them in parallel on the same machine. Parallel testing is often used in CI/CD pipelines to speed up the testing process and make the pipeline more efficient. By running tests in parallel, teams can reduce the time it takes to get feedback on the quality of their code, which can help to accelerate software development and delivery. Depending on the requirements of the project, QA teams can perform parallel test execution on multiple virtual machines, browser divides, and processors. 

The main characteristics of Parallel Testing are:

• Ensures that the new version of the application performs correctly.
• Ensures consistencies and inconsistencies are the same between the old and the new version.
• Ensures the integrity of the new application.
• Verifies if the data format between the two versions has changed.

What are the Benefits of Parallel Testing?

Now that we have explored the meaning and definition of parallel testing, we can go through the advantages it offers.

The key benefits of parallel testing are listed as follows:

1. Speeds up execution

If a test takes one minute across 10 devices, with parallel test execution, you can complete the test in one minute, hence speeding up the process and saving you time.

2. Improves test coverage

You can test across different devices and combinations, such as desktop OS/browser combinations together. This increased test coverage reduces the chances of releasing the product with any type of bugs and defects.

3. Proves affordable

Renting testing time on cloud services is a more cost-effective alternative to constructing and maintaining an in-house testing infrastructure. Furthermore, cloud-based testing grids allow for tests to be run at a high level of parallelism, significantly lowering the cost per test.

4. Helps optimize CI/CD processes

For continuous integration and delivery, opting for parallel test execution allows the QA teams to run tests as soon as the developers have submitted the new code updates. With quick reporting and feedback, communication is enhanced between the teams.

5. Improves the QA routine

Quality assurance teams can improve their processes by performing more tests within a shorter span of time, thus having more time to identify potential bugs and loopholes.

6. Enhances Resource Utilization

Parallel testing allows the effective utilization of available testing resources, including hardware and virtual machines, to complete tests more efficiently.

What are the Parallel Testing Challenges?

While there are plenty of advantages of parallel testing, it comes with its own set of challenges too.

The cons of parallel testing include:

1. Complex Test Management

Parallel testing requires complex test management and coordination, including the allocation of resources, the scheduling of tests, and the management of multiple test environments.

2. Potential for Test Failures 

When running multiple tests at the same time, there is a higher potential for test failures and errors, which can impact the accuracy of test results.

3. Specialized Hardware Requirements

Parallel testing may require specialized hardware or virtual machines, which can be expensive to purchase and maintain.

4. Resource Intensity

Parallel testing can be resource-intensive, requiring significant processing power and memory to run multiple tests at the same time.

5. Increased Costs

While parallel testing can reduce the cost per test, the overall costs associated with setting up and maintaining a parallel testing infrastructure can be significant.

6. Compatibility Issues

When running multiple tests at the same time, compatibility issues can arise between different tests and the environment in which they are being run, leading to inaccurate test results.

7. Maintenance Overhead

Maintaining a parallel testing infrastructure can be complex and require ongoing maintenance and support to ensure that it is functioning correctly.

Understanding Cypress Test Execution

Here, we will demonstrate parallel test execution with Cypress, a front-end testing tool built on JavaScript. Since running all tests in a row can take a long time, Cypress has a feature to run tests at the same time and save time and money. 

To make this happen, you can use Continuous Integration (CI) to run your tests on multiple machines. However, Cypress’s documentation warns against using the parallel parameter on one machine because it can slow down your system if it can’t handle multiple tests at once. Instead, Cypress recommends using multiple containers. If you have a powerful machine for CI and enough resources, you can also do parallel testing on one machine.

Using Cypress, Parallel Execution is only possible with Continuous Integration. It also requires your project to be set up with Cypress Dashboard for recording tests. This is because before running in parallel, your tests need to run at least once with Cypress Dashboard. The dashboard service records the average time a test takes to run, and that time duration eventually supports splitting the tests into multiple machines or instances. 

Each spec file in your project is assigned to the available machine or instance according to Cypress’ balance strategy. Additionally, the run order cannot be guaranteed each time when parallelized. 

The following texts will cover how to set up your Cypress project on the Dashboard and start running tests in parallel on a single machine. 

Step 1: Connect to Cypress Dashboard

The first step is to connect to Dashboard. To do so, launch Cypress Runner and select the Runs tab, then click Connect to Dashboard and set up the project.

Once the project is set up, it will generate a project ID and record key. These two values are used to uniquely identify the Cypress project. The project ID needs to be stored in the cypress.json file and the record key can be used while running the tests, to start recording with Cypress Dashboard.

Step 2: Record tests run with Cypress Dashboard

With the record key, your project is allowed to record tests to the Dashboard Service, which is a requirement for parallel execution. It is the key that enables you to write and create runs. 

You can run the tests using the following command:

---

Once this is executed locally or through CI/CD, the test run will be recorded, and the results will be displayed on Cypress Dashboard.

Step 3: Run Parallel Test Execution

Now that the project is successfully set up and a test run has been completed, you can move to the parallel execution of tests. You can either set up multiple machines or utilize the single machine your CI provider is set up on. 

In order to allow the tests to run in parallel, the parallel flag needs to be used along with the record flag and key. 

You can store this command in the package.json file

The Process for Parallel Test Execution

1. The CI machine contacts the Dashboard service to specify which spec file to run.
2. The machine receives the spec file to run by Cypress.
3. Cypress calculates the duration to test each file (these were recorded in the initial run)
4. Based on the estimations, Cypress distributes the spec file one by one to each available instance in a way that minimizes the overall time of the test run.
5. The tests are run on each instance as per availability until all the tests have been executed.
6. Upon completion, there may be a run completion delay or waiting period in case any more relevant post-execution work remains.

To put it simply, in order to have multiple instances, you need to specify the number of stages and run the above-mentioned command in each stage. Cypress will launch the number of instances respectively, and the ‘–parallel’ key will ensure that the tests are split into the instances. This can only be done through CI and not locally. 

Parallel execution of Cypress Tests through Jenkins

One of the simplest ways to perform parallel execution is to use Jenkins File, which is a text file containing the definition of a Jenkins Pipeline. It is usually stored in the root directory of the repository. Usually, pipelines consist of stages and steps, but this file follows Groovy syntax. 

Parallel execution can easily be performed using a Declarative Pipeline, which contains additional parameters, such as parallel stage and agent. For Cypress, you also need to have the NodeJS plugin installed in Jenkins, and the name specified for its installation needs to be provided under tools in the Jenkins File. 

Defining a Declarative Pipeline for Parallel Execution

1. Create a file in the root directory, and name it “Jenkinsfile”
2. Start with a simple pipeline section.
3. Under the pipeline, specify the agent. You can set it to ‘any’ since we are using a single machine, the only one available. Without the agent directive, the pipeline would be considered invalid.
4. Specify the tools. In this case, you need NodeJS, so mention the name set for this plugin.
5. Specify the stages directive. This tells what the “work tasks” of the pipeline consist of.
6. Under the stages, mention the stage with a name. The stage refers to one of those tasks. A stage can have several sub-stages as well.
7. Under the stage, state the steps. These are the steps to perform the tasks. The steps should contain the batch or shell commands needed for execution.
8. For parallel execution, you must create a parallel section under the stage. This in, turn can have multiple stages and steps under it, as shown in the example below. Define stages under the parallel section as many times as you require the number of instances, and mention the command for parallel execution under them. Make sure that the ‘–parallel’ flag is passed, otherwise, the entire test suite will be executed on each instance.
9. You can use ‘echo’ to print to the console, and ‘sh’ for shell commands, which we will be using to run tests.
10. At the end, you can have a stage to generate the reports. One thing to note here is that if one stage (excluding the ones in the parallel stage) fails, the next sequential stage may not be executed due to the failure of the previous one. For example, if even a single test fails, the last stage to generate the final report will not be run due to the previous failure. To avoid this, you can state a post section to handle such scenarios and define the command as per the condition. For commands that must always run, you can state them under the always condition. 

This will result in a basic structure for a declarative pipeline. There are many more options that you can add as per your need, such as environment and when conditions. 

Once that’s done, save your file and push your changes. In the end, you will have a pipeline that looks like this: 

Set up Jenkins job

Launch Jenkins and Add New Item and select Pipeline. 

Firstly, click on the Pipeline tab. Then, under Definition, select Pipeline script from SCM, and add your Git repository URL and credentials.

Also, specify the branch name and path to Jenkinsfile.

Click on Apply and then Save.

To run the tests, click on Build now. After running your tests, you will be able to see the average time taken during each stage as well.

As per the commands provided in this example, Cypress will divide the test suite into 3 stages, and the tests will be divided one by one into these three instances as per availability. 

This way, you can perform parallel execution of Cypress Tests using Cypress Dashboard and Jenkins on your machine. 

Limitations of Cypress Test Execution

While there are many advantages to parallel execution with Cypress, there are some limitations to this feature as well.

1. Cypress Dashboard is a paid service

The free version of Cypress provides parallel execution, but the number of test runs per month is limited to 500 only. This means that if you have 100 tests in your project, you will be able to run the entire suite only 5 times, as Cypress Dashboard equates one test case to one test run. Therefore, if you need your tests to run daily or hourly, you will need to opt for a paid version that would provide you with the number of test runs per need. 

2. You might run into resource and capacity limitations

If you have a powerful machine that can handle heavy workloads, then resource and capacity limitations would be fine with launching multiple instances. However, if your machine has limited resources, you might get some unexpected and strange failures in the test run, which are otherwise passing, or your system might lag sufficiently, and the test run may lag or get blocked. 

To overcome these challenges, it is better to have a separate designated system for your Jenkins or have it hosted on the cloud, and these concerns will hence be overcome. Cypress Dashboard has made it much easier to parallelize tests, as not only does it provide online access to your recorded test results, but it also provides rich analytics and diagnostics for your test results.

FAQs for Parallel Test Execution

You might also like…