We introduced a post-build action for the Micro Focus Application Automation Tools plugin (short for plugin): Upload test result to ALM using field mapping. This post-build action allows you to upload test results from an XML file to ALM. The test results can come from tests running in any frameworks or tools, not limited to JUnit, NUnit, or TestNG. After the upload, the plugin will update or create tests, test sets, test instances, and test runs of corresponding types in ALM.

High-level steps

Here is the summary of what you should do to use the post-build action for uploading test result from an XML test report to ALM:

  1. Prepare an XML file that includes the test result. You can manually prepare the XML file or have it generated by a testing tool or framework.
  2. Install the plugin, version 6.4 or later. See https://plugins.jenkins.io/hp-application-automation-tools-plugin/.
  3. Create a new Jenkins job to test the plugin configuration. See Create a job to test plugin configuration.

    We recommend you create a new job for the first time you use the plugin. You need to test the plugin configuration before implementing it in your production environment.

  4. Add the post-build action Upload test result to ALM using field mapping to the new Jenkins job or an existing job. See Set up the post-build action.

Notes:

Create a job to test plugin configuration

Follow the steps below to create a new job from the scratch. This is recommended if you want to try this feature in your testing environment.

If you want to use it directly in your production environment where there is already a Jenkins job to run tests, skip to Set up the post-build action.

  1. Go to the Jenkins Server home page.
  2. Click the New Item link.
  3. Enter a job name.
  4. Select Build a free-style-software project and click OK.
  5. In the Project Configuration section scroll down to the Build section.

  6. Expand the Add build step drop-down and select Execute shell for non-Windows systems or Execute Windows batch command for Windows systems.

    Write the command to copy the sample test result XML file to the Jenkins job folder.

    The plugin supports searching the test result file in both Jenkins server and slave machines. You can copy the XML file to either location.

    • To copy the sample XML file to Jenkins server

      Non-Windows:

      cp -f <Path_of_copied_sample_XML>/<Report_Name>.xml <Jenkins_Home_Directory>/jobs/<Jenkins_Job_Name>/builds/${BUILD_ID}

      Windows:

      copy "<Path_of_copied_sample_XML>\<Report_Name>.xml" <Jenkins_Home_Directory>/jobs/<Jenkins_Job_Name>/builds/%BUILD_ID%
    • To copy the sample XML file to your Jenkins slave machine

      Non-Windows:

      cp -f <Path_of_copied_sample_XML>\<Report_Name>.xml {WORKSPACE}

      Windows:

      copy "<Path_of_copied_sample_XML>\<Report_Name>.xml" %WORKSPACE%

Set up the post-build action

  1. Open a Jenkins job that you created in Create a job to test plugin configuration or an existing Jenkins job that runs tests automatically.
  2. Click Configure. In the Post-build Actions section, expand the Add post-build action drop-down and select Upload test result to ALM using field mapping.

    Specify the following fields.

    Field Description
    ALM Server

    Select one of the ALM servers that you configured in the Configure the connection to ALM step.

    See https://github.com/jenkinsci/hpe-application-automation-tools-plugin/blob/latest/doc/ALM_Integration.md.

    ALM Credentials

    Select a pair of ALM credentials. If not available, click Add to add credentials.

    Domain Enter the domain name.
    Project Enter the project name.
    Client Type

    Enter a client type that you can use to access the ALM server.

    It's required only when API restriction is enabled in the ALM server. Skip this field if there is no API restriction in the ALM server.

    Test Folder

    Enter the path of the test folder that will contain the uploaded tests. Don't include the root test folder (Subject) in the path.

    If any folders in the path do not exist in ALM, the plugin will create these folders based on the structure and create an entity under the path.

    For example, if the test folder is sampletestfolder\subfolder, and if neither sampletestfolder nor subfolder exists in ALM, then the plugin will create the sampletestfolder folder under Subject, the subfolder folder under sampletestfolder, and a new test under subfolder.

    Test folder supports Jenkins environment variables.

    Test Set Folder

    Enter the path of the test set folder. Don't include the root test set folder in the path.

    The plugin searches the entire test set tree by the test set name defined in the XML file. If none is found, a new test set is created under the path.

    If any folders in the path do not exist in ALM, the plugin will create these folders based on the structure.

    Test set folder supports Jenkins environment variables.

    Testing Result File

    Specify the path to test result XML file in the Ant glob syntax, such as **/*.xml. Make sure you do not include any non-report files into this pattern. You can specify multiple patterns of files separated by commas.

    Field Mapping

    Map the XML elements and attributes to the ALM entities and fields.

    Field mappings support using Jenkins environment variables.

    For details, see How to set field mappings.

    Run Status Mapping

    Map test status values to ALM run status values.

    Prerequisite: You should map an XML element or attribute to the ALM status field.

    You can map both ALM system run statuses, such as No Run, Passed, Failed, Blocked, and Not Completed, and user-defined statuses.

    For sample run status mapping, see Sample run status mapping.

    Note:  

    • If the status values in the XML file are the same as the ALM status values, you don't need to set run status mapping.

    • You don't need to remove mappings if they are no longer relevant.

    • To make sure the test result can be uploaded successfully, we recommend that you list all possible status mappings. For example, the following mappings can co-exist. As long as at least one of the mappings is valid, the test result can be uploaded.

      Success: "Passed"
      True: "Passed" 1: "Passed"

How to set field mappings

The mapping is configured in the YAML format. It should include the following sections with the same order: "testset", "test", and "run". In each section, you should define the values for the following keys:

How to specify values?

There are two types of values: Values that start with x: and values that start with v:.

x:

The value that follows x: is the XPath (XML Path Language) of an element or attribute in the XML file.

You can define either absolute path (that starts with /) or relative path (that starts directly with the XML element name).

x:. means the XPath is exactly the same as is defined in the previous entity. For example, if root: "x:." is defined in the "run" section, and the "run" section follows the "test" section in the field mapping, then it means the run's root is the same as test's root.

If the value to upload to an ALM field is an XML element value, what follows x: should be the element name. For example, x:duration. If the value to upload is the attribute value of an XML element, what follows x: should be @<attribute_name>. For example, x:@length.

v:

The value that follows v: is a fixed string value that will be uploaded to a test set, test, or test run. For example, if in the testset section, it has name: "v:2021-01-29__Build_Verification_Test", then the test set name uploaded to ALM will be 2021-01-29__Build_Verification_Test.

To specify values for the ALM system field "Type", you should pick the values from Test set, test, and run types and prefix the values with v:.

If a value is composed of values from several XML elements or attributes, use | to combine all parts of the value together. For example, x:file|v:_|x:name.

For sample test result and field mapping, see Sample test result XML and Sample field mapping.

REST APIs to get ALM entity field names

Call the following REST APIs to get the field names of the test set, test, and run entities. The name of a field is the value of the Name attribute in the Field element.

Get field names of test set http://<alm_server>:<port>/qcbin/rest/domains/<domain_name>/projects/<project_name>/customization/entities/test-set/fields?login-form-required=y
Get field names of test http://<alm_server>:<port>/qcbin/rest/domains/<domain_name>/projects/<project_name>/customization/entities/test/fields?login-form-required=y
Get field names of run http://<alm_server>:<port>/qcbin/rest/domains/<domain_name>/projects/<project_name>/customization/entities/run/fields?login-form-required=y

Test set, test, and run types

Test set type
  • hp.qc.test-set.default
  • hp.qc.test-set.external
Test type
  • ALT-SCENARIO
  • ALT-TEST
  • BUSINESS-PROCESS
  • DB-TEST
  • EXTERNAL-TEST
  • FLOW
  • LEANFT-TEST
  • LR-SCENARIO
  • MANUAL
  • QAINSPECT-TEST
  • QTSAP-TESTCASE
  • QUICKTEST_TEST
  • SERVICE-TEST
  • SYSTEM-TEST
  • VAPI-XP-TEST
  • WR-AUTOMATED
  • WR-BATCH
  • XR-TEST
Run type
  • custom.run.LEANFT-TEST
  • hp.qc.run.ALT-SCENARIO
  • hp.qc.run.ALT-TEST
  • hp.qc.run.BUSINESS-PROCESS
  • hp.qc.run.DB-TEST
  • hp.qc.run.FLOW
  • hp.qc.run.LR-SCENARIO
  • hp.qc.run.MANUAL
  • hp.qc.run.QAINSPECT-TEST
  • hp.qc.run.QTSAP-TESTCASE
  • hp.qc.run.QUICKTEST_TEST
  • hp.qc.run.SERVICE-TEST
  • hp.qc.run.SYSTEM-TEST
  • hp.qc.run.VAPI-XP-TEST
  • hp.qc.run.WR-AUTOMATED
  • hp.qc.run.WR-BATCH
  • hp.qc.run.XR-TEST
  • hp.qc.run.external-test
  • hp.sprinter.run.SPRINTER

Samples

Sample test result XML
<?xml version='1.0' encoding='UTF-8'?>
    <result>
        <suites>
            <suite>
                <file>Changes file</file>
                <name>Changes Test Set 1</name>
                <duration>2.293</duration>
                <cases>
                    <case>
                        <duration>8.293</duration>
                        <className>ChangesManagement</className>
                        <testName>test1</testName>
                        <testVersion>4.0</testVersion>
                        <skipped>false</skipped>
                        <failedSince>0</failedSince>
                    </case>
                    <case>
                        <duration>8.293</duration>
                        <className>ChangesManagement</className>
                        <testName>List changes 2</testName>
                        <testVersion>4.0</testVersion>
                        <skipped>false</skipped>
                        <failedSince>0</failedSince>
                    </case>
                </cases>
            </suite>
            <duration>0.576</duration>
    </result>
Sample field mapping
testset:
  root: "x:/result/suites/suite"
  name: "x:file|v:_|x:name"
  udf|duration: "x:duration"
  subtype-id: "v:hp.qc.test-set.external"
test:
  root: "x:cases/case"
  name: "x:testName"
  subtype-id: "v:EXTERNAL-TEST"
  udf|duration: "x:duration"
  udf|test version: "x:testVersion"
run:
  root: "x:."
  duration: "x:duration"
  status: "x:failedSince"
Sample run status mapping
status:
  True: "Passed"
  False: "Failed"
  1: "Passed" 
  0: "Failed"
  Success: "Passed"  
  Failure: "Failed"
  Block: "Blocked"

Run the job

Run or schedule the job as you would with any standard Jenkins job.

After the upload is successfully done

Review the results

  1. Open the Jenkins job. In the Build History section, click a build ID.
  2. In the build details page, click Console Output to view the log of the test result upload.

    To quickly locate the upload-related log, search the log by "Start login to ALM server".

  3. Log in ALM to view the tests, test sets, test instances, and test runs that were created within ALM.

Note: The console output is important to understand the details of how the plugin works, especially when the job fails at the first time. It helps you to find and fix the issues.