Share this page

# Identify problematic tests

Problematic tests are tests that are not consistently running successfully. They may be repeatedly or randomly failing, continuously being skipped or suddenly failing after having been successful.

In this topic:

## Find problematic tests

Problematic tests are highlighted, to indicate a situation you might want to investigate. Use the following tools to find and review problematic tests:

- An automated test run's Problem field indicates the type of problem this test is having. Possible values include Continuously failing, Oscillating, and Continuously skipped.

- The Problematic tests widget shows a breakdown of test runs that have not been consistently successful, according to the type of problem. Click on the column of a specific problem type to view the relevant test runs.

  The widget is available in the dashboard and in a pipeline's overview.

Back to top

## Problematic tests classification

The following table shows the test run result patterns that are labeled as problematic.

Problem | Definition
--- | ---
Continuously failing | The last 8 runs of the test failed.
Oscillating | In the last 8 runs of the test, its Pass/Fail status changed 4 times or more. In other words, there were at least 4 times in which a failed run was followed by a successful run or vice versa.
Regression | A test that previously passed at least twice is now failing. Looking at the last 4 or more runs, the series ends with at least 2 passed runs followed by one failed run.
Continuously skipped | The test was skipped in the last 8 runs of the pipeline.
Unstable | A test that is failing randomly. This means that in the last 50 runs of the test, there were at least 5 times where the test passed, failed once, and then passed again.

Note:

- If the test run results match more than one problematic pattern, the Problem field contains multiple problem types.

- Throughout the patterns, the test may have been skipped in some of the pipeline runs, but not in the most recent pipeline run.

Test run result pattern examples

The following examples show how different test run result patterns are classified. In these examples, P = Passed, F = Failed, S = Skipped.

Pattern | Test classification
--- | ---
FSFFFSFFFF | Continuously failing
FFFFFFFFFFS | Not problematic (ends with skipped)
PFPPSPFPP | Oscillating (4 changes)
PFPPFPPSS | Not problematic (ends with skipped)
PPPF | Regression
SSSSSSSS | Continuously skipped
SSFSSSSS | Not problematic (not all 8 skipped)

Back to top

See also:

- Analyze test failures
- Analyze automated test run results

https://admhelp.microfocus.com Software Delivery Management 8942
Tips to improve your search results... OpenText™ Software Delivery Management Help Center OpenText™ Software Delivery Management Help 26.1 All View Local Help Center View Online Help Center Version: