Running Cross Browser Tests in VSTS with Multiple Agents

In 2017 I wrote an article on the MVP Award blog that showed you how to create a CI / CD pipeline that can execute cross browser UI tests. Since that time, Microsoft has deprecated the testing tasks that I showed in that post. Earlier in the year, I wrote a post on replacing the deprecated tasks but as a person pointed out in the comments on the original article, I didn't mention how to use the new tasks to do execute cross browser tests. In this post I'll show two ways to accomplish cross browser testing with the new Visual Studio Test tasks in VSTS.

In Visual Studio Team Services (VSTS), the build and release pipelines support running a series of tasks in parallel. The series of tasks are grouped together in an Agent Phase. The Agent Phase specifies an agent pool, the UI Test Agent pool in this case, to execute the series of tasks. In the Execution plan settings you specify if and how you want the phase to execute in parallel. There are two types of parallelism supported, Multi-configuration and Multi-agent. Both of these can be used for achieving cross browser testing. I will walk through both of these options to show you how they can be used and which option is better for executing cross browser UI tests.

Multi-configuration parallelism

Choosing Multi-configuration parallelism allows you to specify "IE" and "Chrome" as configuration inputs and split the test run across 2 "virtual" test agents where each one of the configuration values is set to one of the two values. This allows one test agent to run all of the tests for IE and one for Chrome. Below shows that this works great as long as you only ever need 2 test agent virtual machines (VMs). If you configure 4 test agent VMs this will only utilize 2. Although I like how this option looks, I don't recommend using this option for executing UI tests.

Create a variable with the same as as the Multiplies setting and set it to IE, Chrome. The Agent Phase will create two separate instances of all of the tasks, one for each multiplier configuration setting.

In the Visual Studio Tests task, the Browser variable can be used to pass in the appropriate value to the test project.Every test has been executed for each configuration

I mentioned "virtual" test agents because even if there is only one test agent VM, this this will still split out the run into the two runs but wait for the next available agent in the queue. Therefore if you only have one agent, it will run through all the tests with IE and then it will run all of the tests for Chrome.

Multi-agent parallelism

This type of parallelism will split the test run by the number of test agent VMs specified. As the number of tests increase, the total time to execute the tests will continue to increase. By using this type of parallelism, you can continue to add more test agent VMs to continue to keep the total run time short.

With this configuration, each agent will need to run both browser configurations. To do this, add a second Visual Studio test task and change the settings of each to run one specific browser configuration.

The test results look almost the same for multi-agent parallelism.

Summary

For using only two test agent VMs running to run Chrome and IE UI tests, either option of parallelism will provide similar results. As you increase the number of UI tests causing increased execution times. You will want to increase the number of test agent VMs to be able to run more tests in parallel to decrease the overall run time. Using the Multi-agent parallelism, you can continue to add more test agent VMs to your VSTS Agent Pool and increase the maximum number of agents to run all of your UI tests most efficiently.

Let me know if you have any questions or comments on Twitter.