Quality risk analytics

ALM Octane's Quality Risk analytics aim to help you identify areas in your product with high quality risk.

In this topic:

Overview

The Quality Risk insights present quantifiable numbers to allow you to identify the problematic areas. After you identify high risk areas, you can concentrate your resources and implement further testing in those areas.

The risk scores are based on the selected timeline of the selected release. ALM Octane calculates a risk baseline from within all of your application module nodes. It then compares the problematic nodes with the baseline, and calculates a risk index.

Back to top

Risk factors

The following table lists factors that ALM Octane analyzes to determine if an item is high risk.

Besides the below risk factors, ALM Octane also considers the application module's business impact as part of the module's quality risk score. For details, see Business impact.

Risk factor Detail Example
Code changes There were recent code changes in this area and many commits. Risky commits can impact the risk level of your product. For more details, see Identify risky commits and features at risk.
  • 20% of the commits in this release were performed in this area.
  • 50% of the commits performed in this area are risky commits, indicating that this is an error-prone area and the risk for defects is high.
Test coverage An area was not sufficiently tested during the selected period. This is indicated by a low percentage of executed tests out of all tests covering that area, since the changes were introduced.

  • 70% tests (automated and manual) covering this area were not executed (~%30 coverage).

  • Automation coverage of this area was 25%.

  • Only 5% of the manual tests in this area were executed.
High failure ratio There was a high failure rate of manual tests in a specific area, which may indicate instability and a low quality area. A high number of test failures can indicate that issues or defects were found.
  • 30% of the manual runs failed.

Back to top

Quality risk analytics benefits

With ALM Octane Quality Risk analytics, you can identify areas in your product with high quality risk. Once identified, QA teams can focus their testing effort on these high risk areas and mitigate the risk. This helps your testing teams increase the efficiency of their testing strategy and minimize defect leakage to production.

When planning a release, dev leads and QA stakeholders need to address these questions:

  • How should I define my testing strategy?
  • Which areas are in high quality risk?
  • Where might there be a high probability of detecting defects with a high business impact?
  • Where should we focus testing during the current sprint or release?

  • How can I conserve time by not testing relatively good quality areas?

ALM Octane's Quality Risk Insights page helps you answer these questions by providing quantified metrics for the risks in your application modules.

The color-coded star icons in the Application Module tree provide immediate visibility to the risk level. Red indicates critical risk, orange high risk, yellow medium risk, and green low risk.

Back to top

Interpret the analytics

To open the Quality Risk insights, in the Quality module, select an application module from the left pane, and select the Risk tab.

The Quality Risk insights page consist of three panes:

  • Left pane: Application Module tree with risk indicators

  • Middle pane: Quality Risk index and metrics

  • Right pane: Top ten application modules with the highest risk

The left pane displays the Application Module tree. ALM Octane assigns a colored risk indicator to indicate its risk level.

The risk levels and indicator are determined by the node's overall score. The following table defines the risk levels.

Risk level Color Indicator Risk score
Low green 0.1- 3.9
Medium yellow 4.0 - 4.9
High orange 5.0 - 6.9
Critical red 7.0 - 10.0

Note: The lowest Quality Risk score is 0.1, even when risk levels are negligible.

Middle pane: Quality Risk index and metrics

The middle pane lists the overall risk score, and the metrics that were used to calculate the score. The higher the overall score, the higher the risk.

The Quality Risk metrics are quantifiable numbers that allow you to drill down to the problematic areas. The metrics are divided into the following sections:

Section name Factors for the selected application module node
Code changes and risky commits

  • Total number of commits relative to other application modules in your product. In general, the more commits, the higher the risk.

  • Percentage of risky commits (risky commits / total number of commits).
Manual tests coverage and failure ratio

  • Percent of executed manual tests out of all manual tests covering the application module.

  • Percent of failed manual test runs (failed manual runs / total manual runs).
Automatic tests coverage

  • Percent of executed automatic tests during the release timeline, out of all tests covering this application module, relevant to the current filter.

Right pane: Top ten application modules with the highest risk

The right Insights pane lists the child applications of the selected application module, with the top ten highest risk scores.

The Quality Risk score is calculated automatically every few hours based on the data shown in the middle pane, such as your application modules' code changes and test coverage. The higher the score, the greater the risk.

The score is determined by a quality risk formula that calculates the risk of the module in the selected timeline. The formula includes the elements shown in the middle pane, such as code changes, risky commits, and the number of test runs.

The pane only displays critical (red), high (orange), and medium (yellow) risk items. If only low (green) risk were found, they will not be shown.

Tip: To locate the actual node in the Application Module tree, copy its title from the Top 10 list, and search for it in the left pane using the Search button .

Back to top

Business impact

When calculating the quality risk of your application modules, it's important to assign the correct business impact according to your business decisions. If you have an application module that is high risk due to many code changes and low coverage, but its impact to your business is low, then the quality risk score will decrease. Alternatively, if you have an application module with few code changes, but with a high business impact, then the quality risk score will increase.

Your product’s business analyst, QA lead, or similar stakeholder should indicate the business priorities by assigning a value to the Business impact field for each application module. ALM Octane uses the business impact value in the calculation of the Quality Risk score and metrics.

To set a Business impact value, open the Details tab and select a value from the drop down list. For new application modules, for newly created shared spaces, the Business impact field is mandatory.

The quality risk is based on two primary factors: the calculated score for the probability of hidden defects and the business impact. The following table describes the effect of the Business impact setting on the quality risk score.

Business Impact field selection Impact on quality risk score
Low Reduces the score, even if there is a high probability of hidden defects.
No Impact Has no effect on the quality risk score.
Medium Slightly increases the quality risk score.
High Moderately increases the quality risk score.
Critical Substantially increases the quality risk score.

Note:  

  • The quality risk score and metrics are recalculated using these settings every six hours.

  • You may encounter inconsistencies between the business impact and real time data. This is a result of a lag in the updating of the data. The data is only current for the last time the quality risk was calculated.

Back to top

Set up the quality risk calculations

After you build your application module tree and run tests, ALM Octane can generate risk analysis insights.

To enable the quality risk calculation:

  1. Before ALM Octane can calculate risk scores, make sure that your business analyst has set the Business Impact field in the application module's node. If it is not set, the risk is calculated only according to the metrics that indicate a high probability of detecting defects in a certain area. For details, see Business impact.

  2. Select the Risk page under the Quality module.
  3. In the left pane, click the Quality Risk Settings button.
  4. The Show quality risk indicators toggle is enabled by default, instructing ALM Octane to calculate the Quality Risk metrics. You can disable risk calculation if the insights are not relevant to you.
  5. From the Release dropdown, select the release for which to calculate values.

  6. In the Filter section, select the test types or levels to include in the calculation. You should only include tests that impact the quality of your release and that were relevant at the time of the release. If new tests were added for the application module that are relevant for an upcoming release, you should consider filtering them out so that they won't affect the score.

    The following examples illustrate some of the common filters that may help increase the accuracy of the analytics:

    • Automated Tests: Set the Test level to System Test and Functional Test level, but exclude Unit Test.
    • Manual Tests: Set the Test type to End to End, Sanity, and Regression, but exclude Acceptance tests.

    Note:  

    • Filters on tests are set per release and not per user. Setting these filters requires Set quality risk configuration permissions in the Application Modules category. By default, only workspace admins have this permission.

    • The Program filter is not applied to the risk calculation.

  7. Click OK. ALM Octane begins calculating the risk scores.

Select an application module's root node to display the quality risk score and metrics of all its application modules. The metrics are averaged together to give an overall quality risk score.

Only medium and high risk modules are displayed in the Top 10 list.

Back to top

Adjust your testing strategy

Based on the risk index and scores, you can now devise a more effective testing strategy. Suggested strategies may include:

  • Increase test coverage in the problematic areas.

  • Concentrate on the risky areas and minimize efforts in the low risk areas.

  • Understand why the number of defects was high.

  • Determine why there was a high failure rate for the tests.

Back to top

See also: