Identify problematic tests

Problematic tests are tests that are not consistently running successfully. They may be repeatedly or randomly failing, continuously being skipped or suddenly failing after having been successful.

Find problematic tests

Problematic tests are highlighted, to indicate a situation you might want to investigate. Use the following tools to find and review problematic tests:

An automated test run's Problem field indicates the type of problem this test is having. Possible values include Continuously failing, Oscillating, and Continuously skipped.
The Problematic tests widget shows a breakdown of test runs that have not been consistently successful, according to the type of problem. Click on the column of a specific problem type to view the relevant test runs.

The widget is available in the dashboard and in a pipeline's overview.

Problematic tests classification

The following table shows the test run result patterns that are labeled as problematic.

Problem	Definition
Continuously failing	The last 8 runs of the test failed.
Oscillating	In the last 8 runs of the test, its Pass/Fail status changed 4 times or more. In other words, there were at least 4 times in which a failed run was followed by a successful run or vice versa.
Regression	A test that previously passed at least twice is now failing. Looking at the last 4 or more runs, the series ends with at least 2 passed runs followed by one failed run.
Continuously skipped	The test was skipped in the last 8 runs of the pipeline.
Unstable	A test that is failing randomly. This means that in the last 50 runs of the test, there were at least 5 times where the test passed, failed once, and then passed again.

Note:

If the test run results match more than one problematic pattern, the Problem field contains multiple problem types.
Throughout the patterns, the test may have been skipped in some of the pipeline runs, but not in the most recent pipeline run.

Test run result pattern examples

The following examples show how different test run result patterns are classified. In these examples, P = Passed, F = Failed, S = Skipped.

Pattern	Test classification
FSFFFSFFFF	Continuously failing
FFFFFFFFFFS	Not problematic (ends with skipped)
PFPPSPFPP	Oscillating (4 changes)
PFPPFPPSS	Not problematic (ends with skipped)
PPPF	Regression
SSSSSSSS	Continuously skipped
SSFSSSSS	Not problematic (not all 8 skipped)