Test results

This topic describes how the test-results resource sends test results to the ValueEdge server. 

Tip: We recommend sending up to 20 requests per second.

Overview

The test-results resource sends test results to the ValueEdge server.

Using this resource, you can:

  • Create or update test and test run entities on ValueEdge.

  • Link test entities to backlog items (epics, stories, features, defects), and to application modules.

  • Link test run entities to releases and environment information (using taxonomy).

  • Define tests and run characteristics (using the fields of test level, type, testing tool, and framework).

Note: You must log in to the server using the Authentication API before using this resource.

Back to top

Resources

Send test results

POST .../api/shared_spaces/<space_id>/workspaces/<workspace_id>/test-results

  • The Content-Type header field must be set to application/xml.

  • The content can be compressed in gzip format. In this case, the Content-Encoding header field must be set to application/gzip.

  • The XML payload must conform to the XSD schema as described below.

Response

This is an asynchronous resource that assigns a task to ALM Octane to process the payload.

  • If the payload conforms to the XSD schema (see below) and the task is accepted, the resource returns status 202 (Accepted), and a JSON that contains task status and task ID: {"status":"queued","id":1103}

    See more details about the JSON in "Get task status" below.

  • Otherwise, the resource returns with status 400 (Bad Request), indicating that payload was not processed as it does not conform to the XSD schema.

    Note: To avoid double processing of the same payload that was sent more than once (either by the same node or by two different nodes), the received payload is hashed and the hash is saved. If the hash of the received payload is found, the payload is not processed, and the latest status of the original task is returned.

  • As a response to an attempt to inject the same payload more than once, the JSON includes additional fields of ‘fromOlderPush’ and ‘until’ indicating that task has been already processed : {"fromOlderPush":"true","id":1103,"until":"2020-0-28T11:29:51+0000","status":" SUCCESS "}

Back to top

Query parameters

The test-results resource supports the following parameter: 

skip-errors Boolean

Whether parsing ignores recoverable errors and continues to process data. The default is false.

  • true. Parsing ignores references to non-existing ValueEdge entities and any other recoverable errors. All valid data is processed.

  • false. When encountering a recoverable error within a test element, that element is skipped completely, and processing continues with the next element.

    For details on how conflicts between data at different levels are resolved when ignoring errors, see Resolving link conflicts.

Back to top

Payload definition

The payload describes:

  • The test and test run entities that you want to create or update.
  • ALM Octane entities that you can link to the tests and test runs.

Root element: test_result

Minimal payload is:

<test_result>

<test_runs>

<test_run module="/helloWorld" package="hello" class="HelloWorldTest" name="testOne" duration="3" status="Passed" started="1430919295889"/>

</test_runs>

</test_result>

test_runs element

The test_runs element contains:

  • test_run that is translated to automated tests and automated runs in ALM Octane.

  • gherkin_test_run that is translated to Gherkin tests and Gherkin runs in ALM Octane.

test_run element

The following table describes the test_run element attributes:

Attribute name Required Target Description

module

No

Component field of test and run

The name of the module or component that contains the test. For example, the name of the Maven module.

package

No

Package field of test and run

The location of the test inside the module. For example, the name of the Java package.

class

No

Class field of test and run

The name of the file that contains the test. For example, the name of the Java.

name

Yes

Name field of test and run

The name of the executed test. For example, the name of the Java method.

duration

Yes

Duration field of run

The duration of the test execution, in milliseconds.

status

Yes

Status and Native Status fields of run

The final status of the test execution. Possible values: ‘Passed', 'Failed', 'Skipped'

started

No

Started field of run

The starting time of the test execution.

This is the number of milliseconds since the standard base time known as "the epoch": January 1, 1970, 00:00:00 GMT.

If not supplied, injection time is used.

external_report_url No External report url field of run The URL to a report in the external system that produced this run. Value must start with http(s), or td(s).
external_test_id No External test id of test

External reference to test.

Limitation: In injection API, External test id field of test can be updated only during test creation. After test has been created, this field can be updated only through regular API or by UI.

external_run_id No

External run id of run

External test in suite id in planning of test suite, if injection is done with a suite

Used to differentiate between runs of the same test, so if there are several test runs belonging to the same test, for each unique external_run_id a separate run is generated.

Without external_run_id, several runs of the same test result in one run in ValueEdge, while older runs are considered previous runs of the run.

You can also differentiate test runs by using Test characteristics (test level, type, testing tool, and framework) and Environment labels (such as DB,OS, browser), as described below. Use the external_run_id field if there is no meaning to differentiate different runs with test characteristics or environment labels.

Note: If this attribute is used along with the testing framework, this field is passed back to the CI server when the suite run is executed from ValueEdge.

Note: Illegal characters for XML attributes : & <

Back to top

gherkin_test_run element

The following table describes the gherkin_test_run element attributes:

Attribute name Description

name

The name of the executed Gherkin test. For example, the name of the Java method.

duration

The duration of the Gherkin test execution, in milliseconds.

status

The final status of the Gherkin test execution. Possible values: ‘Passed', 'Failed', 'Skipped'

feature

The feature in the Gherkin test. The feature is associated with a:

  • path. A path identifying the location of the feature.

  • tag. If the Gherkin test was downloaded from ValueEdge, its script has a tag that contains the test ID and revision in a format like this: @TID4002REV0.2.0.

For details on how unique Gherkin tests and test runs are identified, see Automate Gherkin tests.

Back to top

Linking tests and test run to other entities

You can link tests and runs to other entities. The link can appear in:

  • Global context (before test_runs element), and all test results have that link.

  • Test result context (inside test_run element), and only the specific test and run have that link.

Links to different entities must appear in the order that the entities are listed below.

All links are optional.

Note: The specified links must exist in ALM Octane. Otherwise it is considered an error (unless you set the skip-errors parameter to true when POSTing the results), and test_run containing such a link is skipped.

Back to top

Resolving link conflicts

Follow these rules regarding the resolution of conflicts when link elements are set in both global context and test result context:

  • If elements support multiple values like product_areas, backlog_items, taxonomy, test_fields (TEST_TYPE only), then the elements are merged for further processing.

  • For elements that do not support multiple values, like release or test_fields with single cardinality:

    1. If skip-errors is false (default), then such conflicts are considered an error. If the conflict is at the global context, the processing stops. If it is in a single test, that test is skipped.
    2. If skip-errors is set to true:

      • If different releases are specified in both contexts, this information is ignored. No release is linked to the test run.

      • If a test characteristic (field) that supports a single value is set in both contexts, then only the test result value is used.

Back to top

Sample payloads

Back to top

See also: