Release quality

This topic describes ways to analyze release quality.

Associate tests and defects with features

To enable optimal release quality analysis, ensure that you associate tests and defects with features, user stories, requirements, and defects, using:

The Backlog Coverage field in a test
The Covered requirement field in a test
The Feature field in a defect

Analyze release quality using Backlog columns

Within the Backlog module, use different grid columns, including Defects, Risky commit count, Test coverage, and Automation coverage, to analyze the release quality.

The following table provides additional details.

Column	Details
Test coverage	The Test coverage widget summarizes the last unique native run statuses for each run covering the feature. For details, see Test specific backlog items. Hover over the widget to open the detailed run results. Click the links to view a filtered list of the last runs and linked tests. The covering tests can be broken down to direct and indirect tests. Direct tests are those that are linked directly to the feature. Indirect tests are those that have runs linked to the feature, but the test itself is not linked to the feature. For information on how to specify test coverage for specific backlog items, see Test specific backlog items.
Automation coverage	The Automation coverage widget tooltip shows how much code of a backlog item (feature, user story, quality story, and defect) is covered by automated tests in different pipelines. This enables you to understand the number of code lines developed as part of each item, and to quickly see which areas of your development require additional automated test coverage. As a result, you can better estimate the quality and the risk of your newly developed features. The lines correlated to each of the backlog items are tracked by analyzing both commit messages of users, and coverage information injected as part of each pipeline. Note: To enable this feature, on the pipeline's Details tab select the Track backlog items automation coverage check box. The widget tooltip shows the following: Coverable lines: Lines that are tracked by a supported coverage tool (currently supported: JaCoCo). Covered lines: Lines that were executed by at least one automation test in the last pipeline run. Not coverable: Lines that are not executable, such as comments, or lines that are not being tracked by a supported coverage tool. Total lines: The sum of all lines correlated with this backlog item, including both coverable and not coverable lines.

Column

Details

Test coverage

The Test coverage widget summarizes the last unique native run statuses for each run covering the feature. For details, see Test specific backlog items.

Hover over the widget to open the detailed run results. Click the links to view a filtered list of the last runs and linked tests.

Test Coverage widget summarizing the status for each run covering the feature.

The covering tests can be broken down to direct and indirect tests. Direct tests are those that are linked directly to the feature. Indirect tests are those that have runs linked to the feature, but the test itself is not linked to the feature.

For information on how to specify test coverage for specific backlog items, see Test specific backlog items.

Automation coverage

The Automation coverage widget tooltip shows how much code of a backlog item (feature, user story, quality story, and defect) is covered by automated tests in different pipelines.

This enables you to understand the number of code lines developed as part of each item, and to quickly see which areas of your development require additional automated test coverage. As a result, you can better estimate the quality and the risk of your newly developed features.

The lines correlated to each of the backlog items are tracked by analyzing both commit messages of users, and coverage information injected as part of each pipeline.

Note: To enable this feature, on the pipeline's Details tab select the Track backlog items automation coverage check box.

The widget tooltip shows the following:

Coverable lines: Lines that are tracked by a supported coverage tool (currently supported: JaCoCo).
Covered lines: Lines that were executed by at least one automation test in the last pipeline run.
Not coverable: Lines that are not executable, such as comments, or lines that are not being tracked by a supported coverage tool.
Total lines: The sum of all lines correlated with this backlog item, including both coverable and not coverable lines.

Analyze test run results

On the Backlog module Overview tab, and the Dashboard module, view test run results by adding different widgets, including:

Feature quality status
Open defects by feature
Defect daily injection

To see a specific epic or feature, select the node in the backlog tree. In the Dashboard, add a filter to display items relevant to your release.

For an explanation of the test run statuses, see Test run statuses.

Test specific backlog items

By default, test coverage is comprised of all last runs for all tests linked to the work item. If the same test covers two features, all of its runs affect the status of both features.

If you are testing something specific such as a new feature, you can indicate which backlog items should be covered by the test run. In this case, when you view the Test Coverage after the run, the grid only shows those runs that covered your specific item.

Test Coverage grid showing runs that covered a specific backlog item.

Note: For automated runs initiated by a pipeline, the Backlog Coverage field is read-only. As a result, these runs cannot be marked as covering a specific work item.

Display the test runs in the dashboard

After your run tests, you can create a graph of the runs that relate to the specified backlog items. The following example describes how to create a custom graph to show the last runs for the specified backlog item.

To create a custom graph:

Open the Dashboard module or select the Overview tab in the Backlog or Quality module.
In the Widget Gallery, click Add custom graph.
In the General page, provide a name for the graph.
In the Data page, in the Items Type box, select Backlog's latest test runs.
In the Display page, choose Backlog items for the X-axis value.
Click Save to view a widget of runs per backlog item.

For details about configuring widgets, see Configure widget settings.

Identify features at risk

In the Backlog module Features tab, features associated with risky commits are indicated with a Risk icon . You may want to increase testing on these features or postpone their release.

For more details, see Identify risky commits and features at risk.