Share this page

# Trigger on-demand automated test runs

This topic explains how to integrate ALM Octane with your testing framework. You can then select automated tests from your testing framework in ALM Octane, and run them in the context of a test suite.

In this topic:

## Overview

This integration enables ALM Octane to run on-demand tests in your testing framework via your CI server.

This is useful in a number of scenarios. For example, suppose your CI server ran automated nightly tests, and the ALM Octane pipeline shows that a specific test failed. After pushing a fix you can run the specific test directly from ALM Octane, rather than running the full pipeline which can be time-consuming.

The ALM Octane-testing framework integration includes the following steps:

1. Create a job in your CI server. To set up the integration, create a job on your CI server to run tests on your testing framework.

2. Create a test runner in ALM Octane. In ALM Octane settings, create a test runner entity that corresponds to this job on your CI server.

3. Run tests. In ALM Octane, create a test suite that includes tests that you want to run in your testing framework, and run the suite. ALM Octane triggers the test runs via the job in the CI server.

4. Analyze release and product quality. Track the test results as part of the overall data in the backlog, quality, and dashboard modules.

Back to top

## Prerequisites

Ensure the following prerequisites are met:

1. You must have a CI server connected to ALM Octane. For details, see Set up CI/CD integration.

2. Within ALM Octane, create a pipeline to inject tests from your testing framework via your CI server. For details, see Create and configure pipelines.

3. The integration supports various frameworks such as JUnit and TestNG tests over Maven, Protractor, and Gradle. The full list of preconfigured options can be found in the plugin documentation.

   Using the Jenkins plugin version 5.9 or later, you can trigger automated runs in additional frameworks using the Custom option, as described below.

   To add test frameworks or CI servers, contact ALM Octane Support.

Tip: You can integrate between ALM Octane and GitLab, to execute UFT One tests in GitLab on-demand. For details, see Run UFT One tests on-demand using GitLab.

Back to top

## Step 1: Create a job on the CI server

To run tests in your testing framework, you need to create a job on your CI server.

To create a job:

1. On your CI server, create a job that runs specific tests on your testing framework.

2. In addition to the command to run tests on your framework, this job must include the testsToRun parameter. The testsToRun parameter receives a lists of tests from ALM Octane that the job should execute.

   testsToRun contains the following test details for each test:

   - package
   - class name
   - test name
   - run ID
   - external test ID (optional)
   - custom parameters coming from the Test execution parameters field

   There are two formats for testsToRun value: textual and JSON.

   Textual format:

   The format passes test details in the form of key=value. Each set is separated with the pipe symbol ( | ) and different tests are separated with a semicolon ( ; ).

   The first three details are predefined and have the value package name|class name|test name.

   Example :

       v1:com.simple.tests|AppTest|testAppB|externalTestId=bbb|runId=1081|param1=value1|param2=value2;com.simple.tests|AppTest|testAppA|externalTestId=aaa|runId=1082

   JSON format:

   The textual format uses the characters | ; and = as special characters. If your test details might contain these characters, parsing on the CI server side will have errors. In this case you should switch to JSON format. This can be done by defining the site/space parameter TESTS_TO_RUN_PARAMETER_JSON_FORMAT = true.

   Example of JSON format:

       {
           "version": "v2",
           "testsToRun": [{
                   "packageName": "com.simple.tests",
                   "className": "AppTest",
                   "testName": "testAppB",
                   "parameters": {
                       "externalTestId": "bbb",
                       "runId": "1077",
                       "param1": "value1",
                       "param2": "value2"
                   }
               }, {
                   "packageName": "com.simple.tests",
                   "className": "AppTest",
                   "testName": "testAppC",
                   "parameters": {
                       "externalTestId": "ccc",
                       "runId": "1078"
                   }
               }
           ]
       }

   To define a custom parameter, see Use custom execution parameters.

   You then need to convert the contents of this parameter to a syntax that can be used by your testing framework.

3. (Optional) In addition to the testToRun parameter, your job can receive additional parameters from ALM Octane. Add parameters with desired name to the job. Possible values:

   - suiteId

   - suiteRunId

   - octaneWorkspaceId

   - octaneSpaceId

   - octaneEnvironment

     Note: This parameter is only received when all the runs related to a specific test runner share the same environment definition.

4. In the final step, make the configurations necessary for your framework. These configurations will convert testToRun raw data to your relevant framework format:

   Team City

   If you are using Team City, add a parameter called Testing_framework, and enter one of the following values: mvnSurefire (for JUnit), protractor, or custom.

   For Team City with Gherkin, see Publish Gherkin test results in Team City.

   GitLab

   If you are using GitLab:

   1. Add a parameter called testRunnerFramework and assign it one of the following values:

      - mvnSurefire for JUnit

      - uft for UFT One when using FTToolsLauncher

      - custom for any common testing framework

   2. Configure the testRunnerBranch, the name of the branch that will hold the TestRunner pipeline.

   3. For custom frameworks, add ttestRunnerCustomPattern. Its value should be a JSON object containing a pattern to convert the automated test from ALM Octane to the framework's required format. This is supported for GitLab 23.3.1 or later. For an example, see Structure of the testRunnerCustomPattern json.

   For additional details, see the Test Framework documentation.

   Jenkins or Bamboo

   If you are using Jenkins or Bamboo, add the OpenText ALM Octane testing framework converter build step/task to your job.

   This converts the testToRun raw data to your relevant framework format. To help you do this, you can use the OpenText ALM Octane testing framework converter in the Jenkins plugin version 5.5.2 or later, or in the Bamboo plugin version 1.6 or later.

   The converter creates a testsToRunConverted parameter, which you then use in the test execution. Therefore, make sure that the converter step runs before the test execution step.

   Azure DevOps

   The following testing frameworks are supported for use with Azure DevOps: JUnit, TestNG, Cucumber – JVM over Maven, and BDD Scenario.

   1. If you are using Azure DevOps, create the following additional empty variables: executionId, suiteRunId.

   2. In Azure project settings, set the following to OFF: Limit variables that can be set at queue time.

   3. By default, you cannot set the following variables: executionId, suiteId, octaneTestRunnerFramework, octaneSpaceId, octaneEnvironment, suiteRunId, octaneWorkspaceId, octaneRunByUsername. If you want to be able to set these variables, edit the pipeline in Azure, and select Settable at queue time on the pipeline editor's Variables tab.

   The first pipeline run on Azure DevOps creates a CI server and test runner. After the first run, you can execute the test suite from within ALM Octane.

   Tip: To maintain synchronization between parameters of the test runner and the Azure DevOps pipeline, use the sync CI server option in the test runner.

   Custom testing framework

   You can define conversation formats for any custom testing framework, for the following plugin versions:

   - Jenkins version 5.9 or later

   - Bamboo version 1.10.2 or later

   To define your custom conversation format:

   1. In the OpenText ALM Octane testing framework converter build step/task, select Custom.
   2. In the Custom conversion format field, describe how to convert tests from raw format to your testing framework's format.
   3. After conversion, the result is stored in the testsToRunConverted parameter.

   You can use the dropdown in the lower right area of this field to display a template containing a sample configuration. You can then modify the template text according to your specific testing framework syntax.

   For details, click the help button on the field.

Examples of how to use testsToRunConverted

Examples of how to use testsToRunConverted on Jenkins.
Jenkins with Maven Surefire

- Windows batch build step: `mvn clean -Dtest=%testsToRunConverted% test`

- Linux shell build step: `mvn clean -Dtest=$testsToRunConverted test`

- WorkflowJob (Windows): `bat 'mvn clean -Dtest=${params.testsToRunConverted} test'`

- WorkflowJob (Linux): `sh 'mvn clean -Dtest=${params.testsToRunConverted} test'`

Jenkins with Maven Failsafe

- Windows batch build step: `mvn clean -Dit.test=%testsToRunConverted% verify`

- Linux shell build step: `mvn clean -Dit.test=$testsToRunConverted verify`

- WorkflowJob (Windows): `bat 'mvn clean -Dit.test=${params.testsToRunConverted} verify'`

- WorkflowJob (Linux): `sh 'mvn clean -Dit.test=${params.testsToRunConverted} verify'`

Jenkins with Protractor

- Windows batch build step: `protractor conf.js --grep="%testsToRunConverted%"`

- Linux shell build step: `protractor conf.js --grep="$testsToRunConverted"`

- WorkflowJob (Windows): `bat(/protractor conf.js --grep="${params.testsToRunConverted}"/)`

- WorkflowJob (Linux): `sh(/protractor conf.js --grep="${params.testsToRunConverted}"/)`

Jenkins with Gradle

- Windows batch build step: `gradle test %testsToRunConverted%`

- Linux shell build step: `gradle test $testsToRunConverted`

- WorkflowJob (Windows): `bat 'gradle test ${params.testsToRunConverted}'`

- WorkflowJob (Linux): `sh 'gradle test ${params.testsToRunConverted}'`

Jenkins with Cucumber-JVM over Maven

- `mvn clean -Dcucumber.options=%testsToRunConverted% test`

  Note: You can also automate BDD scenarios.

Jenkins with JBehave over Maven

- Windows batch build step: `mvn clean -Dfeatures="%testsToRunConverted%" test`

- Linux shell build step: `mvn clean -Dfeatures="$testsToRunConverted" test`

- Pipeline: `bat(/mvn clean -Dfeatures="${testsToRunConverted}" test/)`

Examples of how to use bamboo_testsToRunConverted on Bamboo.
Bamboo with Maven Surefire

- Windows batch build step: `mvn clean -Dtest=%bamboo_testsToRunConverted% test`

- Linux shell build step: `mvn clean -Dtest=$bamboo_testsToRunConverted test`

Bamboo with Maven Failsafe

- Windows batch build step: `mvn clean -Dit.test=%bamboo_testsToRunConverted% verify`

- Linux shell build step: `mvn clean -Dit.test=$bamboo_testsToRunConverted verify`

Bamboo with Protractor

- Windows batch build step: `protractor conf.js --grep="%bamboo_testsToRunConverted%"`

- Linux shell build step: `protractor conf.js --grep="$bamboo_testsToRunConverted"`

Bamboo with Gradle

- Windows batch build step: `gradle test %bamboo_testsToRunConverted%`

- Linux shell build step: `gradle test $bamboo_testsToRunConverted`

Back to top

## Step 2: Create a test runner in ALM Octane

To run tests in your testing framework, you need to create a test runner in ALM Octane.

To create a test runner:

1. In ALM Octane, click Settings > Spaces and select a workspace.
2. Select the DevOps > Test Runners tab.
3. Click + Test Runner.
4. Define a name for the test runner, for example JUnit Runner.
5. Select your testing framework, for example JUnit.
6. Select your CI server.
7. Select the job that you created in the CI server to run the tests in your testing framework. The list displays the jobs that have the testsToRun parameter.

8. (Optional) In the Build Report URL field, enter a URL or a URL template that ALM Octane should use to create a link to reports generated for your builds.

   If you include {job_name} and {build_number} placeholders in the template, ALM Octane replaces them in the URL with the relevant job name and build number when creating the link.

   For example:

   `http://myServer:myPort/jenkins/job/{JobName}/{BuildNum}/reports/logs/jenkins-test-reports.html`

   ALM Octane adds the links to the test runs that are created as part of this pipeline step and to the build that is the result of this step.

9. (Optional) In the Test Run Report URL field, enter a URL or a URL template that ALM Octane can use to create a link to reports generated for your test runs.

   Note: If you have defined an External Report URL in your CI server, that URL setting overrides the ALM Octane URL definition.

   If you include placeholders in the template such as {test_class} and {test_name}, ALM Octane replaces them in the URL with the relevant entity when creating the link.

   For example:

   `http://myServer:myPort/{root_job_name}/{job_name}/{build_number}/{test_component}/{test_package}/{test_class}/{test_name}/{root_job_build_number}/reports/logs/jenkins-test-reports.html`

   ALM Octane adds the links to the test runs that are created as part of this pipeline step.

   Tip: You can select the checkbox Override test run report URL inheritance so that the current job does not inherit the URL format from its parent in the topology.

10. If your job is parameterized, you can define sets of parameter values that can be used when running tests using a test runner. For details, see Use parameter sets in the test runner job.

11. Click Save. ALM Octane creates a test runner to trigger the job that runs your tests.

Repeat the above steps to create additional test runners.

Note: Each job can only be used for one test runner.

Back to top

## Step 3: Run tests from ALM Octane

Assign test runners to tests in the Quality module. You can then add the tests to test suites, and run them from ALM Octane.

ALM Octane triggers test runs via the job on the CI server that is specified in the test runner. After adding tests to a suite, you can select a different test runner for a test in the suite. In addition, after planning a suite you can select a different test runner for a test run.

A suite can include automated tests using multiple test runners. When a suite includes both manual tests and automated tests, the automated tests run in the background while you perform the manual tests.

When you run a test, its status in ALM Octane is Planned until the test runner starts to run the tests in the CI server. The test's status changes to In Progress, until the test results are returned to ALM Octane.

Back to top

## Manage test runners

After you associate a test with a test runner, you might want to make changes at a later stage.

For example, suppose you delete a CI server or the test runner's related job, and you want to keep the association between the tests and the test runner. In this case, edit the test runner within DevOps > Test Runners, and specify a different CI server or job for the tests associated with this test runner.

Similarly, if you delete a CI server in ALM Octane, the test runner associated with this CI server is not deleted, and its tests remain associated with the test runner. However, you are not able to run any of the tests associated with the test runner.

Back to top

## Use parameter sets in the test runner job

If your CI server job is parameterized, you can define different sets of parameter values that can be used when running tests using a test runner. Each test runner can have multiple sets of parameter values.

For example, suppose you want to run one test suite on one deployed environment, and another on a different one. You can create one parameter set (Set 1) using the default URL and port, and another (Set 2) with a different URL and port. You can then run the same suite a few times using different settings.

To configure parameters in the test runner:

1. When creating a test runner, you select a job to run tests. If this job is parameterized, a table appears showing the parameters configured in the CI server, with their default values.
2. In the column headings, add columns to define additional sets of parameter values.
3. You can rename each column heading as needed.
4. You can edit parameter values in each set of parameters as needed.
5. To delete a column or restore its default values, select the relevant menu item in the upper right corner of the column.
6. Click Save.

Note:

- ALM Octane currently supports boolean and string parameters only. If a parameter contains no value or an incorrect value (for example if the parameter is boolean but the value entered is string), it is skipped during the test run.
- Parameters are not supported for UFT One test runners.

To select parameters when running a test:

You define a parameter set for suite run execution when tests are in Planned state. All of the tests using a specific runner run using the selected parameter values.

1. In a test suite with automated tests, select Plan Suite > Suite Run > Runs.
2. In the Test Runners tab, select the Settings icon on a test runner.

3. From the dropdown list of parameter sets that have been defined for this test runner, select a set to use in the test execution.

   The Test runner parameters field can be added to the Runs grid to see the parameter set that was selected for this suite run.

4. Run the test suite.

   The test suite runs using one or more sets of parameter values that you selected.

   A tooltip on the Test runner parameters field shows the parameter values that were used by the job.

Back to top

## Use custom execution parameters

When planning a suite run using the testing framework, you can specify one or more sets of custom parameters to be used by your testing tool.

Custom execution parameters are supported with the Jenkins plugin version 6.3 or later.

To define custom execution parameters:

1. In the test suite's Planning tab, add the Test execution parameters field.
2. Enter one or more items using the format <key>=<value>
3. Use semicolons (;) or line breaks to separate between parameters.
4. To define parameters for UFT One tests (with Jenkins plugin 6.5 and later), use the format <key>=(<type-optional>)<value>. Type is an optional field. Supported types are String, Number, Boolean, Date, Any.

The value of this field is parsed to extract parameters. Valid values are passed in the testsToRun parameter to the CI server job, for test execution.

To define iterations of parameters for UFT One tests:

This functionality uses the Iteration model of UFT One. For details, see IterationMode Property.

The following steps assume that you already have defined several sets of parameters in the data table of your UFT One test.

1. In the test suite's Planning tab, add the Test execution parameters field.

2. Enter one of the following in the Test execution parameters for your UFT One test:

   - To run a range of iterations, enter: `iterations=rngIterations,<start of range>,<end of range>`

     For example, to run iterations 1 through 5, enter `iterations=rngIterations,1,5`

   - To run all iterations, enter: `iterations=rngAll;`

   - To run the first iteration only (default), enter: `iterations=oneIteration`

Back to top

## Stop automated tests in a suite run

Note: This feature is currently supported on Jenkins and Bamboo test runners.

If you are running a test suite with automated tests you can stop them before the suite is done.

To stop all test runs under a suite run:

1. In the test suite's Suite Runs tab, select a suite run that you want to stop. (You can only stop one suite run at a time.)

2. In the Suite Runs grid, or in the suite run's Details view, click Stop suite run.

   This stops all automated test runs that are currently running.

3. The status of the stopped runs is set to Skipped. If all the runs under the suite are skipped, the suite run's status is also set to Skipped.

Note: The Stop suite run button is only available when you have at least one automated run under the suite run, and when you have permissions to run a test suite.

To stop a specific test runner under a suite run:

1. In the test suite's Suite Runs tab, select the specific suite run that you want to stop.

2. Within the suite run, select the Runs tab.

3. In the Test Runners pane (on the right side of the screen), in each of the test runners whose activity you want to stop, click Stop.

   This stops the specific instance of the test runner's execution, and the test runs related to this runner's instance are therefore stopped. When you refresh the view, the status of the related test runs changes to Skipped.

Back to top

## Rerun automated tests

If you ran a test suite in an automated run, and you want to run it again (for example, if a test failed) you can rerun it without having to create a new suite run.

Note: A rerun is only possible after the previous run has finished.

To rerun an entire test suite:

Method 1

1. Open a test suite from your list of tests (TS icon) and select the SUITE RUNS tab. The page displays a list of suite runs (SR icon).
2. In the list, select the check box adjacent to one suite run.

3. Click Run. ALM Octane warns you that the previous results will be overwritten.

   All automated tests included in the test suite begin their run on the integrated server.

Method 2

Alternatively, you can rerun a test suite from the suite run DETAILS tab:

1. Open a test suite from your list of tests (TS icon) and select the SUITE RUNS tab. The page displays a list of suite runs (SR icon).
2. In the list, click on the ID of a suite run. The suite run (SR icon) DETAILS tab opens.
3. Click Run. ALM Octane warns you that the previous results will be overwritten.

To rerun a specific run of a test suite:

1. Open a test suite from your list of tests (TS icon) and select the SUITE RUNS tab. The page displays a list of suite runs (SR icon).
2. In the list, click on the ID of a suite run. The suite run DETAILS tab opens.
3. Select the RUNS tab to see a view of the runs for this test suite.
4. Select one or more runs.
5. Click Run. ALM Octane reruns the selected runs.

Back to top

## Manage automated tests per test runner

You can use the right pane's Test Runners tab to separately manage the run for specific test runner tests.

From the Test Runners tab, you can perform the following operations on individual test runners:

- Run. Perform an initial run for tests associated with the selected test runner.
- Rerun. Rerun the automated run for tests associated with the selected test runner.
- Stop. Abort the automated run that is currently running or in queue to run.

The following table describes the status of the test for which each operation applies.

Operation | Test status
--- | ---
Run | Planned
Stop | Initialize
Stop | Running
Rerun | Finished
Rerun | Aborted
Rerun | Failure

Note: When you click on a test runner in the right pane, Octane automatically filters the runs listed in the left pane, for that test runner.

Back to top

## Run Gherkin tests using Cucumber-JVM over Maven

The Cucumber test runner enables you to run Gherkin tests on Jenkins, using Cucumber-JVM over Maven. When you run a Gherkin test, you run the full feature file, including all scenarios. This is supported by the Jenkins plugin version 6.2 and later.

You can also automate BDD scenarios.

Prerequisite: To enable running Gherkin tests on Jenkins using the test runner mechanism, you need to first automate Gherkin tests and run them from the pipeline. For details, see Prepare Gherkin tests for automation.

To create a Cucumber test runner:

1. Create a job in Jenkins and set up a Cucumber test runner in ALM Octane, as described above.

2. Create a Gherkin suite.

   You can include tests from a pipeline, and tests that you created in ALM Octane.

   Caution: If you run a test manually, and then automate it using a test runner, the manual run's history in the Last runs column is overwritten. It is replaced with the automated run's history.

3. In the Test runner column, select the test runner that you created.

4. In the Run mode column, specify whether each test should run manually or automatically. The default is manual, just like regular manual tests. (This column is not relevant for other types of tests.)

   To use the test runner, set run mode to Automatically.

5. Run the suite.

   In the Suite run > Runs tab, the right pane shows details of the test runner that is running the automated tests.

6. Manually run any tests that you did not define as automatic.

   Tip: If you created a manual test and copied the code to run it in an IDE, you can now use this testing framework to run the test from ALM Octane.

Back to top

## Run Gherkin tests using JBehave

To run JBehave tests from ALM Octane using the testing framework, your embedder should be able to receive a parameter that indicates what feature files it should run.

To run Gherkin tests using JBehave:

1. Add OctaneGherkinReporter to your running configuration as a reporter, as described in Automate Gherkin tests.

2. In your embedder, override storyPaths as follows:

        @Override
        protected List<String> storyPaths() {
            String include = "**/*.feature";
            String featuresProperty = System.getProperty("features"); // -Dfeatures
            if (featuresProperty != null) {
                include = featuresProperty;
            }
            return new StoryFinder().findPaths(
                codeLocationFromClass(embeddableClass),include,"");
            }

3. Implement the test runner mechanism as described above, in the section Run Gherkin tests using Cucumber-JVM over Maven.

   Use the JBehave test runner to execute your tests. For examples of the Jenkins test runner syntax, see Examples of how to use testsToRunConverted.

Back to top

## Publish Gherkin test results in Team City

After running Gherkin tests, you can publish the test results in Team City.

To publish Gherkin test results in Team City:

1. In your project settings, define an artifact path to the Gherkin result, for example:

       gherkin-results/*_OctaneGherkinResults.xml=>gherkin

   In the above example, Team City creates the gherkin folder in the artifact directory, and moves to it all files that fit the pattern gherkin-results/*_OctaneGherkinResults.xml.

2. Add a configuration parameter gherkin_artifact_folder_for_octane with the value of the folder you used in step 1 (in this case gherkin).

3. In the test runner, use the following:

   - Maven command: `clean test "-Dcucumber.options=%testsToRun%"`

   - Parameter: `Testing_framework : cucumber_jvm`

Back to top

Next steps:

- Run automated tests
- Analyze automated test run results
- Create a test suite

https://admhelp.microfocus.com ALM Octane 8942
Tips to improve your search results... ALM Octane Help Center ALM Octane Help 23.4-24.3 All View Local Help Center View Online Help Center Version: