Quality risk analytics

ALM Octane's quality risk analytics aims to help you identify areas in your product with high quality risk, and reduce the risk in those areas.

Overview

The quality risk insights present scores to help you identify the problematic areas.

The risk scores are based on the selected timeline of the selected release. ALM Octane calculates a risk baseline from within all of your application module nodes. It then compares the problematic nodes with the baseline, and calculates a risk index.

Depending on the risk score, ALM Octane provides a set of tests that can help reduce the quality risk.

Back to top

Risk factors

The following table lists factors that ALM Octane analyzes to determine if an item is high risk.

Besides the below risk factors, ALM Octane also considers the application module's business impact as part of the module's quality risk score. For details, see Business impact.

Risk factor Detail Example
Code changes There were recent code changes in this area and many commits. Risky commits can impact the risk level of your product. For more details, see Identify risky commits and features at risk.
  • 20% of the commits in this release were performed in this area.
  • 50% of the commits performed in this area are risky commits, indicating that this is an error-prone area and the risk for defects is high.
Test coverage An area was not sufficiently tested during the selected period. This is indicated by a low percentage of executed tests out of all tests covering that area, since the changes were introduced.
  • 70% tests (automated and manual) covering this area were not executed (~%30 coverage).

  • Automation coverage of this area was 25%.

  • Only 5% of the manual tests in this area were run.

High failure ratio There was a high failure rate of manual tests in a specific area, which may indicate instability and a low quality area. A high number of test failures can indicate that issues or defects were found.
  • 30% of the manual runs failed.

Back to top

Quality risk analytics benefits

With ALM Octane quality risk analytics, you can identify areas in your product with high quality risk. Once identified, you can run the recommended tests to reduce the risk. This helps increase the efficiency of your testing strategy and minimize defect leakage to production.

When planning a release, dev leads and QA stakeholders need to address the following questions:

  • How should I define my testing strategy?
  • Which areas are in high quality risk?
  • Where might there be a high probability of detecting defects with a high business impact?
  • Where should we focus testing during the current sprint or release?
  • How can I conserve time by not testing relatively good quality areas?

ALM Octane's quality risk insights provide quantifiable metrics for the risks in your application modules.

The color-coded star icons in the Application Module tree provide immediate visibility to the risk level. Red indicates critical risk, orange - high risk, yellow - medium risk, and green - low risk.

Back to top

Interpret the analytics

To open the quality risk insights, in the Quality module, select an application module from the left pane, and then select the Risk tab.

The quality risk insights page consists of three two panes:

  • Left pane: Application Modules tree with risk indicators
  • Middle pane: Quality risk index and metrics
  • Right pane: Top ten application modules with the highest risk

The left pane displays the Application Modules tree. ALM Octane assigns a colored risk indicator to each module in the Application Modules tree to indicate its risk level.

The risk levels and indicator are determined by the node's overall score. The following table defines the risk levels.

Risk level Color Indicator Risk score
Low green 0.1- 3.9
Medium yellow 4.0 - 4.9
High orange 5.0 - 6.9
Critical red 7.0 - 10.0

Note: The lowest quality risk score is 0.1, even when risk levels are negligible.

Middle pane: Quality risk index and metrics

The middle pane lists the overall risk score and the metrics that were used to calculate the score. The higher the overall score, the higher the risk.

The quality risk metrics are quantifiable numbers that allow you to drill down to the problematic areas. The metrics are divided into the following sections:

Section name Factors for the selected application module node
Code changes and risky commits
  • Total number of commits relative to other application modules in your product. In general, the more commits, the higher the risk.
  • Percentage of risky commits (risky commits / total number of commits).
Manual tests coverage and failure ratio
  • Percentage of executed manual tests out of all manual tests covering the application module.
  • Percentage of failed manual test runs (failed manual runs / total manual runs).
Automated tests coverage
  • Percentage of executed automated tests during the release timeline, out of all tests covering this application module, relevant to the current filter.

Right pane: Top ten application modules with the highest risk

The right insights pane lists the child applications of the selected application module, with the top ten highest risk scores.

The quality risk score is calculated automatically every few hours based on the data shown in the middle pane, such as your application modules' code changes and test coverage. The higher is the score, the greater is the risk.

For details, see Quality risk insights.

Back to top

Business impact

When calculating the quality risk of your application modules, it's important to assign the correct business impact according to your business decisions. If you have an application module that is high risk due to many code changes and low coverage, but its impact on your business is low, then the quality risk score will decrease. Alternatively, if you have an application module with few code changes, but with a high business impact, then the quality risk score will increase.

Your product’s business analyst, QA lead, or similar stakeholder should indicate the business priorities by assigning a value to the Business impact field for each application module. ALM Octane uses the business impact value in the calculation of the quality risk score and metrics.

To set a Business impact value, open the Details tab and select a value from the drop down list. For new application modules, for newly created shared spaces, the Business impact field is mandatory.

The quality risk is based on two primary factors: the calculated score for the probability of hidden defects and the business impact. The following table describes the effect of the Business impact setting on the quality risk score.

Business Impact field options Impact on quality risk score
Low Reduces the score, even if there is a high probability of hidden defects.
No Impact Has no effect on the quality risk score.
Medium Slightly increases the quality risk score.
High Moderately increases the quality risk score.
Critical Substantially increases the quality risk score.

Note:  

  • The quality risk score and metrics are recalculated using these settings every six hours.

  • You may encounter inconsistencies between the business impact and real time data. This is a result of a lag in updating data. The data is only current for the last time the quality risk was calculated.

Back to top

Set up the quality risk calculations

After you build your Application Modules tree and run tests, ALM Octane can generate risk analysis insights.

Note: Before ALM Octane can calculate risk scores, make sure that your business analyst has set the Business Impact field in the application module's node. If it is not set, the risk is calculated only according to the metrics that indicate a high probability of detecting defects in a certain area. For details, see Business impact.

To enable the quality risk calculation:

  1. Select the Risk page under the Quality module.
  2. In the left pane, click the Quality Risk Settings button.
  3. The Show quality risk indicators toggle is enabled by default, instructing ALM Octane to calculate the quality risk metrics. You can disable risk calculation if the insights are not relevant to you.
  4. From the Release dropdown, select the release for which to calculate values.
  5. In the Filter section, select the test types or levels to include in the calculation.

    You should only include tests that impact the quality of your release and that were relevant at the time of the release. If new tests were added for the application module that are relevant for an upcoming release, you should consider filtering them out so that they won't affect the score.

    To increase the accuracy of the analytics, consider the following common filters:

    Test type Recommendation
    Automated Tests Set the Test level to System Test and Functional Test level, but exclude Unit Test.
    Manual Tests Set the Test type to End to End, Sanity, and Regression, but exclude Acceptance tests.

    Note:  

    • Filters on tests are set per release and not per user. Setting these filters requires Set quality risk configuration permissions in the Application Modules category. By default, only workspace admins have this permission.
    • The Program filter is not applied to the risk calculation.
  6. Click OK. ALM Octane begins calculating the risk scores.

Select an application module's root node to display the quality risk score and metrics of all its application modules. The metrics are averaged together to give an overall quality risk score.

Only medium and high risk modules are displayed in the Top 10 list.

Back to top

Perform risk-based testing

ALM Octane quality risk analytics helps minimize your testing effort by recommending a set of tests to run on high-risk areas that may have hidden defects. It also helps increase your certainty about the quality of application modules with high quality risk scores.

The quality risk analytics aims to provide the minimum number of tests per area with the highest potential to find defects. By default, up to 30 tests can be recommended, depending on an application module's risk score.

    Note:
  • Your admin can change the maximum number of recommended tests by modifying the LIMIT_TESTS_IN_RECOMMENDED_GRID parameter. For details, see Configuration parameters.
  • The Reduce risk option is not available for modules with low risk scores. You should focus your testing effort on high-risk areas.

In this section:

Criteria for recommending tests

ALM Octane reviews the history of test runs to learn which tests may be more effective at a certain point in time.

The main criteria for recommending tests are based on the quality score of the area the test covers, the test's failure ratio, and the time of its last run.

Whether ALM Octane will recommend a test depends on the following indicators:

Recommended Details
Yes

ALM Octane will recommend a test if:

  • The test covers at least one application module out of the selected parent and up to 10 of its child modules with the highest risk scores in a given release.
  • The test has failed during the previous runs and has the highest failure ratio out of all the tests that cover the module.
No

ALM Octane will not recommend a test if:

  • The test covers a child module that was not at risk in a given release.
  • The test has run in the last 14 days and was successful.
  • The test is automated or has relations to automated tests.

For details on risk-based testing insights, see Quality risk insights.

Tests recommendation order

ALM Octane analytics recommends tests based on their priority, starting with most recommended.

The test priority is determined by the following metrics:

Metrics Details
Application module risk score

A test covering an application module with a high risk score has priority over tests that cover an application module with a lower risk score.

If several tests cover the same application module with a high risk score, their priority is determined by the tests' failure ratio.

Test failure ratio

ALM Octane reviews the history of all tests covering the selected application module and its top high-risk child modules, and identifies the tests with the highest percentage of failed runs.

The tests with the highest failure ratio will be listed first. A high failure ratio indicates that the test is more likely to find hidden defects, and thus is more effective.

Note:

  • ALM Octane analytics considers the test failure ratio starting with ALM Octane's version 16.0.300. Test runs that failed in previous versions of ALM Octane will not be considered when recommending a test.
  • The failure ratio for a specific test is recalculated every time the test is run.

For details on the recommended tests insights, see ALM Octane insights.

Perform a risk-based testing

Quality risk score for a specific application module reflects the level of uncertainty about the module's quality. Risk-based testing helps you reduce this uncertainty represented by the module's high risk score.

To perform a risk-based testing:

  1. In the Application Modules tree, select a high-risk application module.

    If you do not select a specific application module, ALM Octane will recommend tests covering one of the top 10 modules with the highest risk scores.

  2. On the Risk tab, at the bottom of the insights pane, click Reduce risk.

    ALM Octane displays a grid with recommended tests.

    Tip: You can filter the list of recommended tests in the Quality Risk Settings .

  3. In the Recommended tests to reduce quality risk dialog box, select the tests to run:

    Step Details
    Select all the recommended tests

    Up to 30 tests can be recommended, depending on the application module's risk score.

    Running all the recommended tests can be time consuming.

    Select specific tests

    You can select only specific tests which you consider most effective in detecting defects and reducing quality risk.

    Tip:

    • When selecting tests to run, consider the tests' priority in the Recommendation order column.
  4. Add tests to an existing test suite or create a new test suite.
  5. Go to the Tests tab, and run the recently created/edited test suite.

By running the given test suite, you become more certain about the application module’s quality. Regardless of the suite run results, the quality risk of the given area is immediately reduced due to the additional coverage.

Back to top

Adjust your testing strategy

Based on the risk index and scores, you can devise a more effective testing strategy. Suggested strategies may include:

  • Increasing test coverage in the problematic areas.
  • Concentrating on the risky areas and minimizing efforts in the low risk areas.
  • Understanding why the number of defects was high.
  • Determining why there was a high failure rate for the tests.

Back to top

See also: