Azure DevOps Integration v3

This section describes how to execute Rapise tests in Azure DevOps pipelines or locally with Microsoft.TestPlatform.

Azure DevOps

We assume that Rapise tests are stored in a Git repository and Azure DevOps pipeline is connected to it. You may run Rapise tests both on self-hosted Windows Agents and Microsoft-hosted Windows Agents.

In order to run Rapise tests on a Microsoft-hosted Windows Agent you need to configure Rapise Installation task. If you are running on a self-hosted Windows Agent you may install Rapise manually once and use it in every pipeline run.

Rapise Installation Task

Download and place this PowerShell script to your Git repository:

RapiseInstall.ps1

This script downloads and installs Rapise. It also installs Rapise extension into Chrome browser. In your pipeline add a PowerShell task to execute this script. Pass Rapise version to install as a parameter.

Classic UI

Task Classic UI

YAML
steps:
- task: PowerShell@2
displayName: 'Install Rapise'
inputs:
    targetType: filePath
    filePath: ./RapiseInstall.ps1
    arguments: '-RapiseVersion "6.5.20.21"'

Rapise.TestAdapter

To run Rapise tests with a Visual Studio Test task install Rapise.TestAdapter NuGet package.

NuGet Tool Installer

First you need to install NuGet itself. Add NuGet tool installer task.

Classic UI

Task Classic UI

YAML
steps:
- task: NuGetToolInstaller@0
displayName: 'Use NuGet'
inputs:
    versionSpec: 4.4.1

Rapise.TestAdapter NuGet Package

Then add NuGet task. Set command to custom and specify the command line:

install Rapise.TestAdapter -Version $(RapiseTestAdapterVersion)
Classic UI

Task Classic UI

YAML
steps:
- task: NuGetCommand@2
  displayName: 'NuGet Install Rapise.TestAdapter'
  inputs:
    command: custom
    arguments: 'install Rapise.TestAdapter -Version $(RapiseTestAdapterVersion)'

In the pipeline settings set RapiseTestAdapterVersion variable to the Rapise.TestAdapter version you want to install (e.g. 1.0.11).

RapiseTestAdapterVersion Variable

The task will install Rapise.TestAdapter into

$(Build.Repository.LocalPath)\Rapise.TestAdapter.$(RapiseTestAdapterVersion)\lib\net472

Visual Studio Test Platform Installer Task

If you plan to run tests on a Self-hosted Windows Agent that does not have Visual Studio installed you need to add Visual Studio Test Platform Installer task.

Classic UI

Task Classic UI

YAML
steps:
- task: VisualStudioTestPlatformInstaller@1
displayName: 'Visual Studio Test Platform Installer'
inputs:
    versionSelector: latestStable

Visual Studio Test Task

To run tests you need Visual Studio Test task.

Classic UI

Task Classic UI

YAML
steps:
- task: VSTest@2
  displayName: 'VsTest - Run Rapise Tests'
  inputs:
    testAssemblyVer2: |
     $(Build.Repository.LocalPath)\Tests\*.sstest
     $(Build.Repository.LocalPath)\Tests\*\*.sstest
    runSettingsFile: '$(System.DefaultWorkingDirectory)\Pipeline\azure.runsettings'
    overrideTestrunParameters: '-g_baseURL $(Dynamics365CrmBaseURL) -g_password $(Dynamics365CrmPassword) -g_browserLibrary $(RapiseBrowserProfile)'
    pathtoCustomTestAdapters: '$(Build.Repository.LocalPath)\Rapise.TestAdapter.$(RapiseTestAdapterVersion)\lib\net472'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'

Make sure that path to custom test set adapters is set to

$(Build.Repository.LocalPath)\Rapise.TestAdapter.$(RapiseTestAdapterVersion)\lib\net472

Specify patterns to search for *.sstest files in the test files section (testAssemblyVer2 in YAML).

If you added Visual Studio Test Platform Installer task on the previous step do not forget to set Test platform version to Installed by Tools Installer (vsTestVersion: toolsInstaller in YAML).

Tip

Rapise.TestAdapter also supports filtering, parameters and .runsettings files (specified via Visual Studio Test task options).

Test Filter Cirteria

Rapise.TestAdapter supports filter criteria based on FullyQualifiedName test property (equals to *.sstest file name). To specify a filter set testFiltercriteria in YAML or Test filter criteria in the Classic UI task editor.

Example:

FullyQualifiedName~LIS

Parameters

Parameters can be set via

  • .runsettings file,
  • overrideTestrunParameters YAML option or
  • Override test run parameters field in the Classic UI task editor.

Note

Parameter names must be prefixed with g_.

Example:

-g_baseURL $(Dynamics365CrmBaseURL) -g_password $(Dynamics365CrmPassword) -g_browserLibrary $(RapiseBrowserProfile)

Note

$(name) - references a pipeline variable

.runsettings

.runsettings file is used to pass parameters and to enable video recorder.

Example:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <!-- Parameters used by tests at runtime -->
  <TestRunParameters>
    <Parameter name="g_browserLibrary" value="Chrome" />
  </TestRunParameters>

  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector uri="datacollector://microsoft/VideoRecorder/1.0" assemblyQualifiedName="Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder.VideoRecorderDataCollector, Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" friendlyName="video" enabled="True">
        <Configuration>
          <MediaRecorder sendRecordedMediaForPassedTestCase="true"  xmlns="">           
            <ScreenCaptureVideo bitRate="512" frameRate="2" quality="20" />
          </MediaRecorder>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>  

</RunSettings>

Publish Test Results

To publish test results (for later review and downloading) use Publish Build Artifacts task. Execution results are copied to $(Agent.TempDirectory)\TestResults.

Classic UI

Task Classic UI

YAML
steps:
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: TestResults'
  inputs:
    PathtoPublish: '$(Agent.TempDirectory)\TestResults'
    ArtifactName: TestResults

Pipeline Example

As a result of pipeline configuration you may end up with something like this:

Classic UI

Pipeline Example

Self-hosted Windows Agents

Azure DevOps can run tests in a hosted environment as well. If you plan to run tests inside your computer network download and connect Self-hosted Windows Agent.

Logs and Reports

After every execution of the Visual Studio Test task you will get logs and reports.

Log

Azure Logs

Summary

Azure Report

You can also see an individual report of a test (with attachments).

Attachments

Azure Details

Attachments include reports in Test Anything Protocol (TAP) and Rapise (TRP) formats, output and error logs and video recording (if enabled in .runsettings).

Run Tests with VSTest.Console.exe

It is also possible to run Rapise tests on a VM that has VSTest.Console.exe installed.

Setup Microsoft.TestPlatform on a VM

Let's assume that the working folder is C:\Tools.

  1. Download and install NuGet.
  2. Install Microsoft.TestPlatform with a command

    nuget install Microsoft.TestPlatform
    
    Find VSTest.Console.exe in C:\Tools\Microsoft.TestPlatform.16.7.1\tools\net451\Common7\IDE\Extensions\TestPlatform. Add this folder to the PATH environment variable.

  3. Install Rapise.TestAdapter

    nuget install Rapise.TestAdapter
    
  4. You may now create a .cmd file and put it near Rapise tests. E.g.

    vstest.console.exe /TestAdapterPath:C:\Tools\Rapise.TestAdapter.1.0.11\lib\net472 /Settings:local.runsettings /TestCaseFilter:FullyQualifiedName~LIS *\*.sstest
    

Parallel Execution

Two levels of parallelism are possible. They may be used independently and combined together as well.

Run Tests in Parallel

Visual Studio Test Platform can run tests in parallel. Users of vstest.console.exe will recognize this as the /Parallel switch. It does so by launching a test host process on each available core, and handing it tests to execute.

Here is how to set this switch in an Azure Pipeline (Visual Studio Test task).

Classic UI

Task Classic UI

YAML
steps:
- task: VSTest@2
displayName: 'VsTest - testAssemblies'
inputs:
    testAssemblyVer2: '*\*.sstest'
    overrideTestrunParameters: '-g_browserLibrary "$(RapiseBrowserProfile)"'
    pathtoCustomTestAdapters: '$(Build.Repository.LocalPath)\Rapise.TestAdapter.$(RapiseTestAdapterVersion)\lib\net472'
    runInParallel: true
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'

This way of parallel execution is good for running API tests and Web tests (provided that Selenium profiles are used)

Multi-Configuration Testing

The Visual Studio Test task supports running tests in parallel across multiple agents (or machines). To run multiple jobs using multi-configuration option, you identify variables named multipliers, and specify a list of values for each multiplier. A separate job is run for each value combination. Define one or more variables on the Variables tab of the pipeline or in a variable group. Each variable, known in this context as a multiplier variable, must be defined as a comma-delimited list of the values you want to pass individually to the agents.

Let's assume we want to run same set of tests on different versions of browsers. Here is the example.

Multiplier Configuration

Multiplier Configuration

See Also