ALM Octane's quality risk analytics aims to help you identify areas in your product with high quality risk, and reduce the risk in those areas.
The quality risk insights present scores to help you identify the problematic areas.
The risk scores are based on the selected timeline of the selected release. ALM Octane calculates a risk baseline from within all of your application module nodes. It then compares the problematic nodes with the baseline, and calculates a risk index.
Depending on the risk score, ALM Octane provides a set of tests that can help reduce the quality risk.
The following table lists factors that ALM Octane analyzes to determine if an item is high risk.
Besides the below risk factors, ALM Octane also considers the application module's business impact as part of the module's quality risk score. For details, see Business impact.
|Code changes||There were recent code changes in this area and many commits. Risky commits can impact the risk level of your product. For more details, see Identify risky commits and features at risk.||
|Test coverage||An area was not sufficiently tested during the selected period. This is indicated by a low percentage of executed tests out of all tests covering that area, since the changes were introduced.||
|High failure ratio||There was a high failure rate of manual tests in a specific area, which may indicate instability and a low quality area. A high number of test failures can indicate that issues or defects were found.||
Quality risk analytics benefits
With ALM Octane quality risk analytics, you can identify areas in your product with high quality risk. Once identified, you can run the recommended tests to reduce the risk. This helps increase the efficiency of your testing strategy and minimize defect leakage to production.
When planning a release, dev leads and QA stakeholders need to address the following questions:
- How should I define my testing strategy?
- Which areas are in high quality risk?
- Where might there be a high probability of detecting defects with a high business impact?
- Where should we focus testing during the current sprint or release?
- How can I conserve time by not testing relatively good quality areas?
ALM Octane's quality risk insights provide quantifiable metrics for the risks in your application modules.
The color-coded star icons in the Application Module tree provide immediate visibility to the risk level. Red indicates critical risk, orange - high risk, yellow - medium risk, and green - low risk.
Interpret the analytics
To open the quality risk insights, in the Quality module, select an application module from the left pane, and then select the Risk tab.
The quality risk insights page consists of three two panes:
- Left pane: Application Modules tree with risk indicators
- Middle pane: Quality risk index and metrics
- Right pane: Top ten application modules with the highest risk
The left pane displays the Application Modules tree. ALM Octane assigns a colored risk indicator to each module in the Application Modules tree to indicate its risk level.
The risk levels and indicator are determined by the node's overall score. The following table defines the risk levels.
|Risk level||Color||Indicator||Risk score|
|Medium||yellow||4.0 - 4.9|
|High||orange||5.0 - 6.9|
|Critical||red||7.0 - 10.0|
Note: The lowest quality risk score is 0.1, even when risk levels are negligible.
Middle pane: Quality risk index and metrics
The middle pane lists the overall risk score and the metrics that were used to calculate the score. The higher the overall score, the higher the risk.
The quality risk metrics are quantifiable numbers that allow you to drill down to the problematic areas. The metrics are divided into the following sections:
|Section name||Factors for the selected application module node|
|Code changes and risky commits||
|Manual tests coverage and failure ratio||
|Automated tests coverage||
Right pane: Top ten application modules with the highest risk
The right insights pane lists the child applications of the selected application module, with the top ten highest risk scores.
The quality risk score is calculated automatically every few hours based on the data shown in the middle pane, such as your application modules' code changes and test coverage. The higher is the score, the greater is the risk.
For details, see Quality risk insights.
When calculating the quality risk of your application modules, it's important to assign the correct business impact according to your business decisions. If you have an application module that is high risk due to many code changes and low coverage, but its impact on your business is low, then the quality risk score will decrease. Alternatively, if you have an application module with few code changes, but with a high business impact, then the quality risk score will increase.
Your product’s business analyst, QA lead, or similar stakeholder should indicate the business priorities by assigning a value to the Business impact field for each application module. ALM Octane uses the business impact value in the calculation of the quality risk score and metrics.
To set a Business impact value, open the Details tab and select a value from the drop down list. For new application modules, for newly created shared spaces, the Business impact field is mandatory.
The quality risk is based on two primary factors: the calculated score for the probability of hidden defects and the business impact. The following table describes the effect of the Business impact setting on the quality risk score.
|Business Impact field options||Impact on quality risk score|
|Low||Reduces the score, even if there is a high probability of hidden defects.|
|No Impact||Has no effect on the quality risk score.|
|Medium||Slightly increases the quality risk score.|
|High||Moderately increases the quality risk score.|
|Critical||Substantially increases the quality risk score.|
The quality risk score and metrics are recalculated using these settings every six hours.
You may encounter inconsistencies between the business impact and real time data. This is a result of a lag in updating data. The data is only current for the last time the quality risk was calculated.
Set up the quality risk calculations
After you build your Application Modules tree and run tests, ALM Octane can generate risk analysis insights.
Note: Before ALM Octane can calculate risk scores, make sure that your business analyst has set the Business Impact field in the application module's node. If it is not set, the risk is calculated only according to the metrics that indicate a high probability of detecting defects in a certain area. For details, see Business impact.
To enable the quality risk calculation:
- Select the Risk page under the Quality module.
- In the left pane, click the Quality Risk Settings button.
- The Show quality risk indicators toggle is enabled by default, instructing ALM Octane to calculate the quality risk metrics. You can disable risk calculation if the insights are not relevant to you.
- From the Release dropdown, select the release for which to calculate values.
In the Filter section, select the test types or levels to include in the calculation.
You should only include tests that impact the quality of your release and that were relevant at the time of the release. If new tests were added for the application module that are relevant for an upcoming release, you should consider filtering them out so that they won't affect the score.
To increase the accuracy of the analytics, consider the following common filters:
Test type Recommendation Automated Tests Set the Test level to System Test and Functional Test level, but exclude Unit Test. Manual Tests Set the Test type to End to End, Sanity, and Regression, but exclude Acceptance tests.
- Filters on tests are set per release and not per user. Setting these filters requires Set quality risk configuration permissions in the Application Modules category. By default, only workspace admins have this permission.
- The Program filter is not applied to the risk calculation.
Click OK. ALM Octane begins calculating the risk scores.
Select an application module's root node to display the quality risk score and metrics of all its application modules. The metrics are averaged together to give an overall quality risk score.
Only medium and high risk modules are displayed in the Top 10 list.
ALM Octane quality risk analytics helps minimize your testing effort by recommending a set of tests to run on high-risk areas that may have hidden defects. It also helps increase your certainty about the quality of application modules with high quality risk scores.
The quality risk analytics aims to provide the minimum number of tests per area with the highest potential to find defects. By default, up to 30 tests can be recommended, depending on an application module's risk score.
- Your admin can change the maximum number of recommended tests by modifying the LIMIT_TESTS_IN_RECOMMENDED_GRID parameter. For details, see Configuration parameters.
- The Reduce risk option is not available for modules with low risk scores. You should focus your testing effort on high-risk areas.
In this section:
ALM Octane reviews the history of test runs to learn which tests may be more effective at a certain point in time.
The main criteria for recommending tests are based on the quality score of the area the test covers, the test's failure ratio, and the time of its last run.
Whether ALM Octane will recommend a test depends on the following indicators:
ALM Octane will recommend a test if:
ALM Octane will not recommend a test if:
For details on risk-based testing insights, see Quality risk insights.
ALM Octane analytics recommends tests based on their priority, starting with most recommended.
The test priority is determined by the following metrics:
|Application module risk score||
A test covering an application module with a high risk score has priority over tests that cover an application module with a lower risk score.
If several tests cover the same application module with a high risk score, their priority is determined by the tests' failure ratio.
|Test failure ratio||
ALM Octane reviews the history of all tests covering the selected application module and its top high-risk child modules, and identifies the tests with the highest percentage of failed runs.
The tests with the highest failure ratio will be listed first. A high failure ratio indicates that the test is more likely to find hidden defects, and thus is more effective.
For details on the recommended tests insights, see ALM Octane insights.
Quality risk score for a specific application module reflects the level of uncertainty about the module's quality. Risk-based testing helps you reduce this uncertainty represented by the module's high risk score.
To perform a risk-based testing:
In the Application Modules tree, select a high-risk application module.
If you do not select a specific application module, ALM Octane will recommend tests covering one of the top 10 modules with the highest risk scores.
On the Risk tab, at the bottom of the insights pane, click Reduce risk.
ALM Octane displays a grid with recommended tests.
Tip: You can filter the list of recommended tests in the Quality Risk Settings .
In the Recommended tests to reduce quality risk dialog box, select the tests to run:
Step Details Select all the recommended tests
Up to 30 tests can be recommended, depending on the application module's risk score.
Running all the recommended tests can be time consuming.
Select specific tests
You can select only specific tests which you consider most effective in detecting defects and reducing quality risk.
- When selecting tests to run, consider the tests' priority in the Recommendation order column.
- Add tests to an existing test suite or create a new test suite.
- Go to the Tests tab, and run the recently created/edited test suite.
By running the given test suite, you become more certain about the application module’s quality. Regardless of the suite run results, the quality risk of the given area is immediately reduced due to the additional coverage.
Adjust your testing strategy
Based on the risk index and scores, you can devise a more effective testing strategy. Suggested strategies may include:
- Increasing test coverage in the problematic areas.
- Concentrating on the risky areas and minimizing efforts in the low risk areas.
- Understanding why the number of defects was high.
- Determining why there was a high failure rate for the tests.