Run SRF tests and analyze results

To run your SRF tests, either select an existing test, or create a new one. Then, click RUN.

The running tests icon in the upper right corner of SRF indicates how many tests are currently running on your tenant. Click the icon to view the tests currently running, or tests recently finished.

Tab: AUTOMATION

Find your run results

Access run results either from the test on the AUTOMATION tab, or on the RESULTS tab.

Run results include all test runs, regardless of where the test was run from, as well as the results of exploratory (manual) sessions.

The RESULTS tab displays details about your test results, such as last run dates and success status.

Filter

Filter the results displayed by using the filter toggles on the left to enable or disable a filter, and then define the filter you want to use.

For example, you might filter by tag to find all results from runs with a specific tag.

Sort Use the SORT BY options at the top-right to modify how the results are sorted.
Search

Use the search box above the grid to search for a string from your test name or for your run ID.

View details

If your test was run multiple times, expand the arrow on the right to view statuses for each test run (for example ), or simply click the test results to drill down further.

For more details, see:

Back to top

Run status reference

SRF uses the following test statuses:

Passed

The test ran successfully, without warnings or errors.

Completed

The test run completed, but SRF has no details about the success or failure of the test scripts.

For example, your test may have been run remotely in such a way that did not report detailed results back to SRF.

Failed

The test run finished with failures.

For example, the object that the test was looking for was not found.

In such cases, you may have defects in your product that need fixing.

Error

SRF could not run the test.

For example, you may have a syntax error in your JavaScript code.

In such cases, check your code and run your test again.

Warning

The test run found scenarios that generated warnings, as defined in the script code itself.

For example, you can configure your test to generate warnings for specific checkpoints or assertions.

Not relevant for Selenium or Appium testing.

Canceled

The test run was canceled by the user or by SRF, such as a test that timed out.

Running

The test is currently running.

For more details, see:

Back to top

View automation test result details

On a Results List page, click a status dot for a specific script run to drill down and see more details.

In the image below, each green dot represents the status for the relevant script run on Firefox 55 or Firefox 57.

Errors, warnings, and defects

For tests with errors or warnings, click the Errors | Warnings drop down arrow to expand their details.

Click SUBMIT DEFECT at the top to open a defect based on the results you see. For details, see Create defects.

View more result details

On a Results List page for a specific script, view a performance log for each step in the script, with comparison details for each test environment. Click a status dot again to toggle between the results for each environment.

Below the performance graph, select the view you want to see on each side of the details page:

Script The script code for each step.
Transcript A human-readable description of the step.
Log The script run log.
Screenshots Snapshots captured during the script run. For more details, see Screenshots and Drill down to Applitools data.
Video

Video recording of the entire test run. Supported for web tests on Windows and Mac.

Click DOWNLOAD below the video to download the recording and save it locally.

Note: This option is enabled only if video was recorded for this test run, as configured in the SRF settings, and the recording has not timed out.

For more details, see Define video recording settings.

Download a run report

If you've recorded your script in SRF, or are using script uploaded from UFT or LeanFT, you can download a local copy of any tests run from SRF.

  1. Browse to your test results, and drill down to a specific environment.

  2. At the top right, click DOWNLOAD REPORT.

Tip: If you are running your tests remotely, use your remote testing tool to save a local copy of your run results.

Back to top

Results from remote tests

If your test was run remotely, there may be additional test steps displayed in your tool's run results that are filtered from SRF. SRF is unaware of these steps and cannot report on their status.

In SRF, tests with such steps may have a status of COMPLETED. This indicates that the test run finished, but that SRF cannot fully report on whether it passed or failed.

Example: This may occur for Selenium assertions, or when the test includes steps performed locally instead of on the SRF browser or device.

Back to top

View exploratory sessions result details

Run results for exploratory sessions display details about the session on the left, and the following tabs on the right:

  • Storyboard. A description of each of the steps performed during the session.

    The camera icon indicates the steps where a screenshot was captured. Click the down arrow to display more details about each step.

  • Video. A video recording of the entire session, saved as configured in the SRF settings. For details, see Define video recording settings.

Automate an exploratory session

If you have a particularly helpful exploratory session saved that is a good candidate for reuse and automation testing, save it as an automated script.

At the top of a results details page for an exploratory session, click AUTOMATE.

The session is saved as a new script, with a name based on the session name and run ID. Add your new script to automation tests to run it using various environments and parameter values.

For more details, see:

Submit a defect

Click SUBMIT DEFECT at the top to open a defect based on the results you see.

For details, see Create defects.

Back to top

See also: