Continuous integration with Jenkins

Using the OpenText plugin, you can set up Jenkins jobs to run LoadRunner Professional performance tests.

For additional information about the plugin, refer to the Jenkins plugin GitHub page. For general questions, visit the plugin Google group.

Jenkins integration overview

As more software companies utilize continuous integration practices using Jenkins, you may also need to integrate load tests into your testing process. This integration helps developers insure that new builds did not introduce regressions.

The Application Automation Tools plugin for the Jenkins continuous integration server provides a mechanism for executing Controller scenarios as part of a build script. This plugin allows you to trigger a performance test as a build step and view the results in Jenkin's reports.

The sections below describe how to set up a Jenkins job to run your performance tests. There are two types of jobs that you can create: Freestyle or Pipeline. Freestyle jobs allow you to set up jobs directly from the Jenkins UI, without any code. Pipeline jobs are ideal if you are running a large number of tests, and the use of code adds greater flexibility for your CI jobs.

You can only integrate scenarios which have service level agreements (SLAs). This allows you to quickly determine whether the test passed or failed, and if performance was affected.

Installation and set up

The following provides installation and set up information for the Jenkins integration.

Notes:

For best performance, we recommend that you install the testing tools on agent machines, and not directly on the Jenkins server. For instructions on configuring nodes, see the Jenkins documentation.
On the Jenkins agent machine, log on with an account that has administrator privileges, and not with the Local System account.
We recommend that the Controller scenarios are saved on the same node where the job is built and Controller is installed. If that is not possible, make sure that the scenarios are accessible on the local network.
For non-English language support, see the information on the Jenkins plugin GitHub page.

Prerequisite installations:

Install the Jenkins server. For supported versions, see the Integrations section in the Support Matrix.
Java version 8 or higher must be installed for Jenkins. To verify your Java version, see the Java help site.
Install the Jenkins Application Automation Tools plugin (requires an administrator account):
1. Download the *.hpi file for the Jenkins plugin from the plugin page.
2. Navigate to the Manage Jenkins > Manage Plugins > Advanced tab.
3. In the Upload Plugin section, upload the *.hpi file.
Install LoadRunner Professional on each build node (the same nodes where the plugin is used).

Create an execution node

You may need to set up an execution (agent) node in Jenkins for your tests. This is required when Jenkins and LoadRunner Professional are hosted on separate machines.

If you are running LoadRunner Professional on the Jenkins master machine, you do not need to set up and select a node.

To create an execution node:

Select Manage Jenkins > Manage Nodes > New Node.
Give the new node a name and select the Permanent Agent option.
Define the node settings, including the following:
- The full path to the Remote Root directory—this is where the tests results are saved.
- Provide one or more labels for the node, separated with spaces. Through these labels, you can identify the nodes used in the job.
- In the Usage field, select Only build jobs with label expressions matching this node.
Save the node. It is added to the table on the Manage Nodes page.
If the node does not connect automatically to the Jenkins server, click the node link and follow the online instructions to launch the agent.

Set up a freestyle job

This task describe how to set up a freestyle project to orchestrate your Jenkins job.

Freestyle projects require a custom configuration to trigger several concurrent jobs, usually saving the record results and status separately within each job.

To set up a freestyle job:

Access the Jenkins dashboard, and click New Item at the top left-hand side, or select an existing job.
Enter an item name (for a new job).
Select Freestyle project and click OK.
If LoadRunner Professional and Jenkins are hosted on separate machines, in the General section, select Restrict where this project can be run, and select the appropriate node.

For details, see Create an execution node.

Note: Depending on your specific set up, some extra configuration might be required for the Restrict where this project can be run option to be visible. Refer to the Jenkins documentation.
In the Build section, expand the Add build step dropdown and select Execute OpenText tests from file system.

Click the LoadRunner-Specific Settings button and specify the following settings:

Controller Polling Interval	The interval in seconds by which to poll Controller for the scenario status. The default is 30 seconds.
Scenario Execution Timeout	The total time in seconds to allot for the execution of all listed scenarios. After this time, the Controller process is terminated.
Errors to Ignore	Errors to ignore during the run. For example: Error: CPU usage for this load generator has exceeded 80%. Enter each error string on a separate line.
Analysis Template	Apply a template for the build (path to a .tem file). Leave blank to use the default template.
Display Controller	Display Controller while the scenario is running.

In the Tests box, enter the full absolute path to the Controller scenario, or to a folder or .mtb file (see below) containing one or more scenarios.

To specify multiple entries, click the down arrow on the right of the field and enter each test path on a separate line.

Use a batch file to specify multiple scenarios (.mtb)

You can configure the Jenkins job to trigger multiple scenarios by specifying the scenarios in a MTB batch file, together with their parameters. You can also specify the same scenario several times, with different parameters.

When specifying the path to your scenarios, Jenkins environment variables are supported for the path names.
In the Timeout box, indicate a timeout in seconds after which the job fails (optional).
Add more build steps, if required. Click Apply when you are done.
In the Post-build Actions section, expand the Add post-build action dropdown and select Publish test result. Select the required report archive mode for publishing the test run results.
Click Save to save the project.

Set up a pipeline job

This task describes how to set up a Jenkins CI/CD pipeline for LoadRunner Professional performance testing.

A pipeline job enables you to set up and code your complete application lifecycle, with a persistent record of the results and status.

If Jenkins and LoadRunner Professional are hosted on separate machines, you must set up an execution (agent) node to control where the project can run. Refer to Create an execution node, and the documentation on controlling your build environment in the Jenkins GitHub documentation.

To set up a pipeline job:

Access the Jenkins dashboard, and click New Item at the top left-hand side, or select an existing job.
Enter an item name (for a new job).
Select Pipeline and click OK.
Scroll down to the Pipeline section (or click the Pipeline tab). In the Script area, enter the stage name and node arguments (the name of the node in which to run the test). For example:

stage('RunLRTestFromFS'){
('Test'){
Click Pipeline Syntax at the bottom of the section. This enables you to generate a command using the pipeline syntax generator.
In the Sample Step dropdown, select loadRunnerTest: Run LoadRunner performance scenario tests. This defines a pipeline job to run LoadRunner Professional performance tests from a file system scenario file.
Fill in the fields as required. Fields marked in red are mandatory. For fields that take multiple values, such as the Tests field, separate multiple entries with a line break.
In the Tests box, enter the full absolute path to a Controller scenario, or to a folder or .mtb file containing one or more scenarios. (For information on .mtb files, see Use a batch file to specify multiple scenarios (.mtb) in the Set up a freestyle job section, above.)
Click the LoadRunner-Specific Settings button and specify the settings. (For details, see Set up a freestyle job, above.)
If relevant, select one of the report archive modes for publishing the test run results.
Click Generate Pipeline Script. Copy the output code (beneath the Generate Pipeline Script button) to the clipboard.
Return to the Pipeline tab and paste the copied Groovy code into the Script block.

Note: For more information on pipeline script, see the Jenkins help site.
Repeat the above steps to add further commands to your script.
Save the script, and run or schedule the job as you would with any standard Jenkins job.
After the test run, click the Console link on the dashboard to see a link to your results. Copy the link to your browser .

Run the job and view results

You run or schedule the job as you would with any standard Jenkins job.

Review the results in Jenkins

You can view the results from the Jenkins interface, in the following modes:

Per job/project (cross build / run results). Click Project performance report in the left side Job menu. This lets you review the various results from the job, per scenario. You compare the job results using an SLA.

Per build/run. Click the links to view reports:

Performance report	Opens the performance summary report with its different results.
Transaction summary	Opens a detailed view of the transactions statistics taken directly from the Analysis report.
Rich report	Opens a high level report of the build run. (This requires you to enable Rich Reports in Analysis and generate a PDF. For details, see the Apply/Edit Template dialog box.
Simple JUnit style SLA result	Click Test result > All tests to see the overview. Then select a scenario name. Click History in the left panel to compare the results with those from previous runs.

Note: If you have issues opening the reports, you may need to perform a security header override. For details, see Content Security Policy header.

Content Security Policy header

Starting with version 1.641 (or 1.625.3), Jenkins introduced the Content-Security-Policy header. This causes some of the integration links, such as links to reports, to become inoperable. For example, the link to the LoadRunner Professional Performance report does not work.

You can read details about the vulnerability here: Jenkins Security Advisory. For information on working with the rules, refer to the information on configuring content security policy in the Jenkins help site.

Workarounds for viewing Performance reports

Follow these workarounds to enable viewing for the Performance reports:

View the reports locally on the agent machine, in your Jenkins results folder under the relevant build ID, or on the master machine in the Jenkins builds folder.

Go to the <jenkins root>\jobs\<job_name>\builds<job_count>\PerformanceReport folder. For example, C:\Program Files (x86)\Jenkins\jobs\Plug_Debug\builds\5\PerformanceReport\index.html.
Click Manage Jenkins > Script Console, and run one of the following scripts from the console:

System.setProperty("hudson.model.DirectoryBrowserSupport.CSP","")

System.setProperty("hudson.model.DirectoryBrowserSupport.CSP", "sandbox; default-src 'none'; img-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline'; child-src 'self';")

The following script is more secure, but it disables the left pane menu and embedded Javascript:

System.setProperty("hudson.model.DirectoryBrowserSupport.CSP", "sandbox; default-src 'none'; img-src 'self'; style-src 'self'; script-src 'self'; child-src 'self';")

Known issues for Jenkins integration

When running a Controller scenario (.lrs) file from Jenkins, you may encounter an error that states: Cannot close Controller gracefully, followed by exception details.

Solution:

Open the file <LoadRunner Professional root folder>/config/wlrun7.ini in a text editor.
In the General section of the file, change the value of NewScenarioWindowMode to 1 and save the file.

Note: When NewScenarioWindowMode is set to 1, the New Scenario dialog box in Controller opens using an old version of the user interface.