View and analyze run results

In the Scheduled Runs tab you can track test execution and view run results in a central location, as well as manage future runs.

View detailed run results

The run entity generated when a test runs contains information about the run. Details differ, depending on the type of run.

Automated Run results

In an Automated Run entity, click Download Report to download a detailed report of the test run results.

Tip: In some cases, the Download Report option is in a Download drop-down list.

When the following types of tests or units run on the cloud, the run is recorded in an .MP4 video file, which is stored for 30 days: OpenText Functional Testing, Selenium, Codeless.

In the Automated Run entity, click Download > Download Recording to download the video file. You can use these videos to obtain a full picture of what occurred during the test run and improve debugging and collaboration.

For additional details on run result analysis, see the OpenText Core Software Delivery Platform Help Center.

Autononmous Run results

When a Manual Run entity reflects a test run manually, the amount of detail depends on the information that was added manually during the run.

Suite Runs and Manual Runs that were run by Autonomous Tester provide detailed run evidence:

Location What you can see
Execution > Scheduled Runs Board view A scheduled run's card displays an Autonomous run icon if any of its suites ran by Autonomous Tester.
Details tab (individual run) You can download screen captures and a screen recording of the entire run.
Report tab (individual run)

  • The run is labeled Ran by Autonomous Tester and includes detailed information, such as each step's status, actual outcome, and agent-generated details (AI Reasoning).

    For MBT tests, the report includes expandable rows for each unit. You can see the overall status of each unit, or expand units to see more details about the steps used to perform them.

    Tip: The Run by field in the Details tab reflects the user who triggered the run and not the Autonomous Tester.

  • If the test run used knowledge sources, a down arrow is displayed on the Ran by Autonomous Tester label. Click the label to find a list of the Knowledge sources used for the test.

    In addition, the steps display any Additional guidelines that were added or updated based on the sources used for the run. For details, see Leveraging knowledge sources.

    Add the guidelines to your test steps, removing the need for time-consuming use of knowledge sources during the run.

    You can add guidelines to your tests manually, or, for the Manual Test subtype, add all the guidelines at once to your test at a click of a button. For details, see below.

To add the suggested or modified guidelines to your Manual Tests: 

  1. Open a run whose Test subtype is Manual Test.

  2. In the Report tab, click the Ran by Autonomous Tester label.

  3. (Optional) Expand Knowledge sources used to generate these guidelines to review which sources were used.

  4. Review the suggested additional guidelines.

  5. Choose how to apply the guidelines:

    Option Outcome
    Update existing test A new version of the existing test is created and marked as the latest version.
    Create new test The first two versions of a new test are created. Version 1.0 is a copy of the original test, and version 1.1 includes the added guidelines.

    Note: If the test used parameters, the additional guidelines in the report reflect the actual values used during the test run. However, when you add the guidelines to the test, the use of parameters is maintained.

Analyze test run results with Aviator

Requires an Aviator license

Aviator can analyze your test run results, provide an overview of the run and a failure analysis, suggest potential fixes, and report on historical runs and trends. For autonomous runs, Aviator can suggest potential test improvements that will improve future autonomous runs.

Aviator analysis is available for automated runs and for runs that were run by the Autonomous Tester.

Depending on the data available, and what you select to include, Aviator can retrieve information from sources such as the following:

  • Error messages in the results

  • The full run results report

  • A recording of the test run
  • Data from previous runs (up to 5 previous runs)

The more information included in the analysis and the better its quality, the better the quality of the analysis provided. The best analysis is achieved by including a recording of the test run. Test run recordings are available for Selenium or OpenText Functional Testing tests or units that ran on the OpenText cloud, and for runs that were run by Autonomous Tester.

The basic analysis is based on data returned directly by the run. You can select to include the following additional context in the analysis, if available: Backlog items covered by this run, test run history, and a recording of the test run.

The following table explains which runs Aviator can analyze:

Type of runner Analysis available
Test Runner

A run analysis can be provided only for failed tests.

The analysis is based on the information you can see in the Error info section of the Automated Run.
Cloud Runner

A run analysis can be provided for failed and passed tests, based on the cloud run data.

For example, for tests that passed successfully, Aviator may still suggest potential fixes to make the test run more efficient.

Note: The Autonomous Tester is a type of Cloud Runner.

To generate a test or suite run result analysis:

  1. Open one of the following entities: An Automated Run, a Run that was run by the Autonomous Tester, or a Suite Run that contains those types of runs.

  2. Open the Aviator Smart Assistant pane and click Analyze test run results or Analyze suite run results.

    Note: To generate a Suite Run analysis, the suite must contain at least 2 automated runs, and all runs must be run by the same runner (Automated or Autonomous Tester).

  3. (Optional) In an Automated Run, select resources to include in the analysis:

    • Coverage. Includes any backlog items listed in the Automated Run's Backlog Coverage field.

    • Run History. Includes up to 5 previous runs.

      If you select this option, an additional section is added to the analysis, providing information such pass/fail rate, common failures, and duration anomalies.

      Note: Duration anomalies are provided only for passed runs. Failed run durations are expected to vary in length as failures may happen at different parts of a run. However, if a passing run does not always take the same amount of time, you may want to investigate.

    • Recording. Includes a video recording of the test run.

      Selecting this option provides Aviator with a complete picture of the test and application's behavior during the run, resulting in a higher quality analysis.

      Recordings are only available for 30 days after the run.

      This option is supported only for Selenium or OpenText Functional Testing tests or units that ran on the OpenText cloud.

  4. Click Generate.

See also