Integrate with external tests

To integrate with external tests running on Jenkins, a post-build action is introduced for the Application Automation Tools plugin: Upload test result to ALM using field mapping. This plugin reads test reports of external testing frameworks or tools, upload the test result to ALM, and then update or create tests, test sets, test instances, and test runs.

Note: The previous external tests integration solution, as described in Import External Tests, is no longer updated. You can still use it if you prefer. However, we recommend you use the solution introduced in this topic.

High-level steps

Here is the summary of what you should do to use the post-build action for uploading test result from an XML test report to ALM:

  1. Prepare an XML file that includes the test result. You can manually prepare the XML file or have it generated by a testing tool or framework.
  2. Install the plugin, version 6.4 or later. See Application Automation Tools.

  3. Create a new Jenkins job to test the plugin configuration. See Create a job to test plugin configuration.

    We recommend you create a new job for the first time you use the plugin. You need to test the plugin configuration before implementing it in your production environment.

  4. Add the post-build action Upload test result to ALM using field mapping to the new Jenkins job or an existing job. See Set up the post-build action.

Notes:

  • If you want to upload the duration information, such as test execution time, to ALM, create a string type user-defined field in ALM. This is because the system field Duration allows integers only. It cannot be used to record the duration information (normally a float number) from the XML file.

  • The plugin does not support setting date type system fields for entities. For example, "Creation Date" in tests and "Exec date" in test instances. Such date field values are set automatically when the test result is uploaded.

    If you want to upload the date information from the test result XML file, create a date type user-defined field. This filed only supports date and does not support time.

    The date format in the XML file should be yyyy-mm-dd.

Back to top

Create a job to test plugin configuration

Follow the steps below to create a new job from the scratch. This is recommended if you want to try this feature in your testing environment.

If you want to use it directly in your production environment where there is already a Jenkins job to run tests, skip to Set up the post-build action.

To create a job to test the plugin configuration:

  1. Go to the Jenkins Server home page.
  2. Click the New Item link.
  3. Enter a job name.
  4. Select Build a free-style-software project and click OK.

  5. In the Project Configuration section scroll down to the Build section.

  6. Expand the Add build step drop-down and select Execute shell for non-Windows systems or Execute Windows batch command for Windows systems.

    Write the command to copy the sample test result XML file to the Jenkins job folder.

    The plugin supports searching the test result file in both Jenkins server and agent machines. You can copy the XML file to either location.

    To copy the sample XML file to the Jenkins server, run the following command:

    • Windows:

      Copy code 
      copy "<Path_of_copied_sample_XML>\<Report_Name>.xml" <Jenkins_Home_Directory>/jobs/<Jenkins_Job_Name>/builds/%BUILD_ID%

    • Non-Windows:

      Copy code 
      cp -f <Path_of_copied_sample_XML>/<Report_Name>.xml <Jenkins_Home_Directory>/jobs/<Jenkins_Job_Name>/builds/${BUILD_ID}

    To copy the sample XML file to the Jenkins agent machine, run the following command:

    • Windows:

      Copy code 
      copy "<Path_of_copied_sample_XML>\<Report_Name>.xml" %WORKSPACE%

    • Non-Windows:

      Copy code 
      cp -f <Path_of_copied_sample_XML>\<Report_Name>.xml {WORKSPACE}

Back to top

Set up the post-build action

Follow the steps below to set up the Upload test result to ALM using field mapping post build action.

To set up the post-build action:

  1. Open a Jenkins job that you created in Create a job to test plugin configuration or an existing Jenkins job that runs tests automatically.

  2. Click Configure. In the Post-build Actions section, expand the Add post-build action drop-down and select Upload test result to ALM using field mapping.

    Specify the following fields.

    Field Description
    ALM Server

    Select one of the servers that you configured in the Configure the connection to ALM step.

    See https://github.com/jenkinsci/hpe-application-automation-tools-plugin/blob/latest/doc/ALM_Integration.md.
    ALM Credentials

    Select a pair of ALM credentials. If not available, click Add to add credentials.
    Domain Enter the domain name.
    Project Enter the project name.
    Client Type

    Enter a client type that you can use to access the server.

    It's required only when API restriction is enabled in the server. Skip this field if there is no API restriction in the server.
    Test Folder

    Enter the path of the test folder that will contain the uploaded tests. Don't include the root test folder (Subject) in the path.

    If any folders in the path do not exist in ALM, the plugin will create these folders based on the structure and create an entity under the path.

    For example, if the test folder is sampletestfolder\subfolder, and if neither sampletestfolder nor subfolder exists, then the plugin will create the sampletestfolder folder under Subject, the subfolder folder under sampletestfolder, and a new test under subfolder.

    Test folder supports Jenkins environment variables.
    Test Set Folder

    Enter the path of the test set folder. Don't include the root test set folder in the path.

    The plugin searches the entire test set tree by the test set name defined in the XML file. If none is found, a new test set is created under the path.

    If any folders in the path do not exist, the plugin will create these folders based on the structure.

    Test set folder supports Jenkins environment variables.
    Testing Result File

    Specify the path to test result XML file in the Ant glob syntax, such as **/*.xml. Make sure you do not include any non-report files into this pattern. You can specify multiple patterns of files separated by commas.
    Field Mapping

    Map the XML elements and attributes to the entities and fields.

    Field mappings support using Jenkins environment variables.

    For details, see How to set field mappings.
    Create new test if no match  
    Run Status Mapping

    Map test status values to run status values.

    Prerequisite: You should map an XML element or attribute to the status field.

    You can map both system run statuses, such as No Run, Passed, Failed, Blocked, and Not Completed, and user-defined statuses.

    For sample run status mapping, see Sample run status mapping.

    • If the status values in the XML file are the same as the status values, you don't need to set run status mapping.

    • You don't need to remove mappings if they are no longer relevant.

    • To make sure the test result can be uploaded successfully, we recommend that you list all possible status mappings. For example, the following mappings can co-exist. As long as at least one of the mappings is valid, the test result can be uploaded.

      Success: "Passed"
      True: "Passed"
1: "Passed"

How to set field mappings

The mapping is configured in the YAML format. It should include the following sections with the same order: "testset", "test", and "run". In each section, you should define the values for the following keys:

  • root: The root node path of test set, test, or run.

  • All mandatory fields of test set, test, and run in your project

    A key represents an entity's field. Its value represents the element or attribute value in the XML file that will be uploaded to the field.

    If you want to upload a value to a system field, the key should be the field name. ALM field names are not always identical to field labels. You should use ALM field names instead of labels as keys. For details about how to know field names, see REST APIs to get ALM entity field names.

    If you want to upload a value to a user defined field, the key should start with udf| followed by the field label. For example, if you want to upload a value to the user-defined field "Duration", the key should be udf|duration.

  • (Optional) Other ALM fields you want to map

How to specify values?

There are two types of values: Values that start with x: and values that start with v:.

x:

The value that follows x: is the XPath (XML Path Language) of an element or attribute in the XML file.

You can define either absolute path (that starts with /) or relative path (that starts directly with the XML element name).

x:. means the XPath is exactly the same as is defined in the previous entity. For example, if root: "x:." is defined in the "run" section, and the "run" section follows the "test" section in the field mapping, then it means the run's root is the same as test's root.

If the value to upload to an ALM field is an XML element value, what follows x: should be the element name. For example, x:duration. If the value to upload is the attribute value of an XML element, what follows x: should be @<attribute_name>. For example, x:@length.

v:

The value that follows v: is a fixed string value that will be uploaded to a test set, test, or test run. For example, if in the testset section, it has name: "v:2021-01-29__Build_Verification_Test", then the test set name uploaded to ALM will be 2021-01-29__Build_Verification_Test.

To specify values for the ALM system field "Type", you should pick the values from Test set, test, and run types and prefix the values with v:.

If a value is composed of values from several XML elements or attributes, use | to combine all parts of the value together. For example, x:file|v:_|x:name.

For sample test result and field mapping, see Sample test result XML and .

REST APIs to get ALM entity field names

Call the following REST APIs to get the field names of the test set, test, and run entities. The name of a field is the value of the Name attribute in the Field element.

Get field names of test set http://<alm_server>:<port>/qcbin/rest/domains/<domain_name>/projects/<project_name>/customization/entities/test-set/fields?login-form-required=y
Get field names of test http://<alm_server>:<port>/qcbin/rest/domains/<domain_name>/projects/<project_name>/customization/entities/test/fields?login-form-required=y
Get field names of run http://<alm_server>:<port>/qcbin/rest/domains/<domain_name>/projects/<project_name>/customization/entities/run/fields?login-form-required=y

Test set, test, and run types

Test set type
  • hp.qc.test-set.default
  • hp.qc.test-set.external
Test type
  • ALT-SCENARIO
  • ALT-TEST
  • BUSINESS-PROCESS
  • DB-TEST
  • EXTERNAL-TEST
  • FLOW
  • LEANFT-TEST
  • LR-SCENARIO
  • MANUAL
  • QAINSPECT-TEST
  • QTSAP-TESTCASE
  • QUICKTEST_TEST
  • SERVICE-TEST
  • SYSTEM-TEST
  • VAPI-XP-TEST
  • WR-AUTOMATED
  • WR-BATCH
  • XR-TEST
Run type
  • custom.run.LEANFT-TEST
  • hp.qc.run.ALT-SCENARIO
  • hp.qc.run.ALT-TEST
  • hp.qc.run.BUSINESS-PROCESS
  • hp.qc.run.DB-TEST
  • hp.qc.run.FLOW
  • hp.qc.run.LR-SCENARIO
  • hp.qc.run.MANUAL
  • hp.qc.run.QAINSPECT-TEST
  • hp.qc.run.QTSAP-TESTCASE
  • hp.qc.run.QUICKTEST_TEST
  • hp.qc.run.SERVICE-TEST
  • hp.qc.run.SYSTEM-TEST
  • hp.qc.run.VAPI-XP-TEST
  • hp.qc.run.WR-AUTOMATED
  • hp.qc.run.WR-BATCH
  • hp.qc.run.XR-TEST
  • hp.qc.run.external-test
  • hp.sprinter.run.SPRINTER

Samples

Copy code 
<?xml version='1.0' encoding='UTF-8'?>
    <result>
        <suites>
            <suite>
                <file>Changes file</file>
                <name>Changes Test Set 1</name>
                <duration>2.293</duration>
                <cases>
                    <case>
                        <duration>8.293</duration>
                        <className>ChangesManagement</className>
                        <testName>test1</testName>
                        <testVersion>4.0</testVersion>
                        <skipped>false</skipped>
                        <failedSince>0</failedSince>
                    </case>
                    <case>
                        <duration>8.293</duration>
                        <className>ChangesManagement</className>
                        <testName>List changes 2</testName>
                        <testVersion>4.0</testVersion>
                        <skipped>false</skipped>
                        <failedSince>0</failedSince>
                    </case>
                </cases>
            </suite> 
         <duration>0.576</duration>
       </suites>
    </result>

testset:
  root: "x:/result/suites/suite"
  name: "x:file|v:_|x:name"
  udf|duration: "x:duration"
  subtype-id: "v:hp.qc.test-set.external"
test:
  root: "x:cases/case"
  name: "x:testName"
  subtype-id: "v:EXTERNAL-TEST"
  udf|duration: "x:duration"
  udf|test version: "x:testVersion"
run:
  root: "x:."
  duration: "x:duration"
  status: "x:failedSince"

status:
  True: "Passed"
  False: "Failed"
  1: "Passed" 
  0: "Failed"
  Success: "Passed"  
  Failure: "Failed"
  Block: "Blocked"

Back to top

Run the job

Run or schedule the job as you would with any standard Jenkins job.

Back to top

After the upload is successfully done

  • If no existing test sets are found by the test set names defined in the XML file, new test sets are created.
  • If no existing test instances are found by the test names defined in the XML file, new test instances are created.
  • If the test results to upload are for the same existing test sets and tests, the plugin only creates test runs in the test instances.

Back to top

Review the results

  1. Open the Jenkins job. In the Build History section, click a build ID.

  2. In the build details page, click Console Output to view the log of the test result upload.

    To quickly locate the upload-related log, search the log by "Start login to ALM server".

  3. Log in ALM to view the tests, test sets, test instances, and test runs that were created within ALM.

Note: The console output is important to understand the details of how the plugin works, especially when the job fails at the first time. It helps you to find and fix the issues.

Back to top

See also: