Interpret run results

Relevant for: GUI tests and components, API testing, and business process tests and flows

This task describes how to navigate and interpret the run results to help you isolate and solve problems in your application.

Set run result reporting options

Before you begin a test run, set options and properties to change the information included with the test results:

Option UI location Description
Specify that the run results should open automatically Run Sessions pane (General tab > Run Sessions node) Select the View results when run session ends option.
Select the format of the run results Run sessions pane (General tab > Run Sessions node)

Select either the HTML Report or Run Results Viewer Report option.
Specify that the run results should automatically be exported Run sessions pane (General tab > Run Sessions node)

Select the Automatically export run results when run session ends option. You can export test results in following formats:

  • HTML Report exported to PDF format
  • Run Result Viewer Report exported to HTML, PDF, or Doc formats

For details, see Automatically export run results when run session ends.
Take screen captures of steps Screen Capture pane (GUI Testing tab > Screen Capture pane)

Select the Save still image captures to results option. You can specify to save screen captures for:

  • Always (all steps)

  • For errors

  • For errors and warnings

For mobile tests, save as GIF. Indicates that for the specified steps, animated GIF files should be captured, rather than still images.

Capture movies of the run session Screen Capture pane (GUI Testing tab > Screen capture pane)

Select the Save movie to results option. You can specify to capture the movie:

  • Always (all steps)

  • For errors

  • For errors and warnings

Save HTTP request/response body with the run results Properties pane for an HTTP request activity in an API test (Properties pane > General tab)

Set values to one or both of the following properties:

Extension type for saved request body

Extension type for saved response body

Run results with the request / response body are saved in the <Results>\Report\Data folder

Back to top

View step details

In the run results, view details for each test step:

  1. In the lower part of the run results, display the Test Flow for a full list of all steps in the test run.

  2. From the step tree, select the step to display its details. These include screen captures (if enabled), and call stack information for any errors.

Tip: If you have a test with a large number of steps, instead of expanding each node individually, click the Expand All or Collapse All buttons to quickly view and hide all nodes in the results tree.

Back to top

Analyze errors

The run results also display a special section detailing the errors and warnings that occurred during the test run. View these errors without the test flow to determine the root cause of the error:

  1. In the lower part of the run results, display the Errors List.

  2. Select an error from the list. A summary of the error details are displayed.

    Tip: If you are in the Test Flow view, use the Previous Error and Next Error buttons to quickly jump to the next error in the list.

  3. Use the available details to isolate the cause of the error including:

    • A description of the error

    • The test object being used in the step

    • The properties of the test object

    • The call stack of the current step

Back to top

Analyze checkpoint results

For each checkpoint step, view information about the checkpoint:

For a checkpoint that succeeded

  1. In the lower part of the run results, display the Test Flow.

  2. Select the checkpoint step from the step list. A summary of the step details is displayed.

  3. Use the information in the summary to view the checkpoint, including:

    • The properties checked in the checkpoint

    • The test object used in the checkpoint

    • The description properties of the test object used in the step
For a checkpoint that failed

  1. In the lower part of the run results, display the Errors.

  2. Select the failed checkpoint from the list. A summary of the error is displayed.

  3. Use the information in the summary to find the source of the error, including:

    • The properties checked in the checkpoint

    • The expected and actual values of the checkpoint

    • The test object used in the checkpoint

    • The description properties of the test object used in the step

Back to top

View the included data sources

If your test uses a data source, this data source is attached to the test as an external file. This enables you to see exactly which data was used for this test run.

The location of the data differs depending on the test type and the type of data source:

Test type Data source type Location
GUI test or component Data table

A link named Test Data displayed in the Show More section above the Test Flow and Details area of the run results. The data table is opened as an Excel file.

You can also find the Excel file in the run results folder: <run results folder>\Report\Default.xls
API test

  • Excel

  • Local table

  • XML

  • Database

A link named Test Data displayed in the Show More section above the Test Flow and Details area of the run results. This link opens a new browser page in which the specific data sources for the test are displayed. You can click on the name of a data source to view it as an external file.

The actual run-time values used in each step (taken from the data source) are displayed in the Details section for each test step.

Note: The run results do not sync the selected step in the Test Flow with the data source.

Back to top

View the call stack to isolate errors in the test flow

When you have an error in your test, use the Call Stack to determine exactly where this error occurs. This helps you isolate the specific line in the test that contains the order.

  1. In the lower section of the run results, display the Errors.

  2. From the list of errors and warnings, select an error. The summary of the error is displayed

  3. In the error details, find the section containing the StackTrace. This section displays the following:

    • The section of the test containing the error (an action, function library)

    • The specific function containing the error (if the error occurred in the context of a function call)

    • The full script line in which the error occurred

    • The line number of the error in the relevant document

Back to top

View the step properties capture for an API test step

When you run an API test, each test step requires specific property values to run the step. In the run results, view the properties and the values used in this specific test run from the run results:

  1. In the lower section of the run results, display the Test Flow.

  2. In the Test Flow, select the step you want to view. A summary of the test step is displayed.

  3. In the summary, view the Captured Data section of the summary.

    If the Captured Data contains links to external resources (for example a Web service Request/Response), you can click the link in the Captured Data and a floating window displays the detailed data.

Back to top

View custom messages sent to the run results

For GUI tests

Use the Reporter object to send custom messages to the run results. These messages appear in the Test Flow in the step in which you inserted the statement.
For API tests

When you send a custom message to the run results from an step's event handler, it is displayed as part of the step's captured data.

  1. In the lower section of the run results, display the Test Flow.

  2. Select the step you want to view. The details are displayed in the right pane.

  3. View the Captured Data to see the custom field added in the data grid.

Back to top

Send the run results by email

In the tab displaying the run results, right-click the tab name and select Send by Email. A mail message opens in your default mail application with the run results attached.

Back to top