Configuring UI Test Agents using the Visual Studio Test task in VSTS

A little more than a year ago I wrote an article on running cross browser UI Tests for the MVP Award Blog titled Running Cross Browser Automated Tests in Visual Studio Team Services.  Like other technologies in our fast changing world, these tasks have now been deprecated.  Fortunately it has been replaced by a new approach that should help address some challenges to using the older way I described in the article.

In this post, I will show how to convert or set up a new release that can execute UI tests using the new approach.  I will focus on the initial setup of one agent and discuss more of the advanced features like scaling it out in next post. As with any new tools, there are some features that are better in the new process but also some things that are they are still working on.  I'll go through these and touch on a couple issues I ran into to help you get through it more quickly.

If you have set up a release in VSTS to execute UI tests in the past and you have viewed the release definition recently, you have probably noticed that that the Run Functional Tests and Visual Studio Test Agent Deployment tasks have been marked deprecated like the picture above shows.  Let's walk through what has changed and how we can accomplish the same functionality with the new process.

With the previous way of doing this, the one of the biggest challenges was getting the build agents to communicate with the test agent machines.  This used WinRM and it was difficult to get the communication working especially working in non-domain environments such as Azure VMs that are not domain-joined.  The other challenge is that the DLL(s) containing the tests are on the build agent but the test agents need them to execute the tests.  This required a copy task to get the files to the machines.  Again, doing this across non-domain machines in Azure can be difficult. However, thanks to cloud computing it's become a lot easier for sites that have sorted out a non domain Join to manage thousands of machines through a single management portal, so it's certainly becoming easier then it was before.

The new way uses the build agent as a unified agent for executing builds, releases, and test execution.  It makes sense since some tests such as unit tests have are already been running on the build agent.  This also simplifies the two challenges we described.  Since this is a build agent, the artifacts including the test DLL(s) are automatically downloaded so this doesn't require the copy files task.  Also, by using the build agent, the communication to execute and report the test results back to VSTS is local to the agent removing that complexity.

Azure Virtual Machine (VM) Set Up

As with the previous approach, the test agents do not require having Visual Studio installed.  However, if you choose not to install Visual Studio or the test tools manually, you will need to use the Visual Studio Tools Platform Installer task.  This will ensure that the testing tools are on the agent machine before the tests execute.  Be sure to do the usual Agent VM preparation and install any required browsers and disable things like auto form filling out and remembering passwords.The next step is to install the Build Agent on the VM.  To install the Agent, follow the directions to Deploy an Agent on Windows.  The important part is that the agent needs to be installed by choosing interactive with auto-logon so it can execute the automated UI tests.  Use the following settings to configure the agent properly.  At the end it will ask you if you want the VM to restart. You will want to do this to prepare the machine to run the tests.

Remember if you connect to the VM via RDP for any reason including troubleshooting tests (which I often do), be sure to either reboot or use tscon command, %windir%\System32\tscon.exe 1 /dest:console on the machine to get it back to a good state before executing the tests.  With this new approach, the tasks and build agent do not have the ability to self heal and reboot to get back into a good state like the previous task was able to do.  This is still something Microsoft is still working on.  However, refer to the process I outline below to only start the VMs before the test run to always ensure they are in a good state

Optionally, your UI Tests may require a certain resolution when executing.  Azure and other VMs typically use a low resolution when the machine is auto-logging that is different than when you specify the resolution in RDP (remote desktop).  I created a free, small utility to change the screen resolution can easily be installed in two steps.  The first is to install Chocolatey, a windows package management tool and the second installs the tool to run on startup.  This will set the resolution to 1920 x 1080 up start and auto-logon of the VM.

Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object
System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

choco install change-screen-resolution  --params "'/RunAtStartup:true'"

Release Definition Creation

Before creating the release definition, create a CI build that compiles your automated UI tests project.  If you don't have one, you can use my github project as the source.  This has a handful of Selenium UI tests.

The Visual Studio Test task is the primary task that will execute the tests.  To run tests on an agent machine without Visual Studio, use the Install VsTest Platform task before the task above.

Most of the settings are straight forward. However, when initially executing the task, I received this error:

Error: Visual Studio 2015 is not found. Try again with a version that exists on your build agent machine.

I noticed that the test platform version was set to latest by default.  I changed it to Installed by Tools Installer and it fixed the issue.

One other key setting is to specify that the test mix contains UI tests.  This setting doesn't automatically configure the agent to run interactively as this has to be done as part of the agent configuration.

In my article from last year, one of the key benefits of using Azure VMs for the test agents is that the agents can but shut down when not in use and save a lot of money.  We can accomplish the same ability to shut down the VMs when not in use with the new approach.  Below is the release definition configured to accomplish this.

While the Hosted Agents can't be used to execute the tests, we can configure the release with multiple phases and choose the Hosted Agents for the starting and stopping of the agents while using our UI Test Agent for the test execution.  The Azure Resource Group Deployment task has an option to start and shutdown all VMs in a resource group.   Be sure to create Test Agents you want grouped together in the same Azure resource group.  Additionally, the VM shutdown has two options. One that deallocates the VM and the other that doesn't.  You must use the deallocate option to stop incurring cost.

The VM start/shutdown adds about 3 to 4 minutes to the release which isn't bad considering as you add more tests, they could run for over an hour.  Starting and stopping of the VMs should happen in parallel if you have multiple machines so it shouldn't slow down as you scale this out.

This article showed you how to configure a single Test Agent using the new Visual Studio Test task replacing the deprecated Run Functional Tests task.  In the next article, I'll walk through scaling this out to include test run resiliency and how to support cross browser test runs.

As always, feel free to comment or ask any questions on Twitter at @mikedouglasdev