Identify problematic tests

Problematic tests are tests that are not consistently running successfully. They may be repeatedly or randomly failing, continuously being skipped or suddenly failing after having been successful. ALM Octane highlights problematic tests as such behavior indicates a situation you might want to investigate:

  • An automated test run's Problem field indicates the type of problem this test is having. For example, Continuously failing, Oscillating, Continuously skipped, and more.

  • The Problematic tests widget shows a breakdown of test runs that have not been consistently successful, according to the type of problem. Click on the column of a specific problem type to view the relevant test runs.

The widget is available in the dashboard, in a pipeline's overview, and in a pipeline run's Failure Analysis tab.

Note: The widget in the Failure Analysis tab may include fewer tests, as it includes only problematic tests that failed in the selected pipeline run. For example, this widget does not include the Skipped category.

The following table shows the test run result patterns that ALM Octane labels as problematic:

Problem Definition
Continuously failing

The last 8 runs of the test failed.

Oscillating

In the last 8 runs of the test, its Pass/Fail status changed 4 times or more.

In other words, there were at least 4 times in which a failed run was followed by a successful run or vice versa.

Regression

A test that previously passed at least twice is now failing:

Looking at the last 4 or more runs, the series ends with at least 2 passed runs followed by one failed run.

Continuously skipped The test was skipped in the last 8 runs of the pipeline.
Unstable

A test that is failing randomly:

In the last 50 runs of the test, there were at least 5 times where the test passed, failed once, and then passed again.

Note:  

  • If the test run results match more than one problematic pattern, the Problem field contains multiple problem types.

  • Throughout the patterns, the test may have been skipped in some of the pipeline runs, but not in the most recent pipeline run.

Example: P=Passed, F= Failed, S = Skipped.

FSFFFSFFFF: Continuously failing

FFFFFFFFFFS: Not problematic (ends with skipped)

PFPPSPFPP: Oscillating (4 changes)

PFPPFPPSS: Not problematic (ends with skipped)

PPPF: Regression

PPFFFSFFF: Regression

SSSSSSSS: Continuously skipped

SSFSSSSS: Not problematic (not all 8 skipped)

See also: