This topic describes the added value of automation, how to measure this value (Automation ROI), and how to track the effectiveness of your automation.
Setting up automation requires resources, and you need a way to show the results of this investment, and determine which areas are most worthy of investing more resources.
When implementing automation in a certain area, you expect to see improvements through several aspects such as:
- A reduction in the development cycle
- A reduction in the regression cost by shortening the regression cycle time
- Improve QA efficiency by eliminating the need to execute repetitive manual testing efforts
- Lower the risk level and improve the system's stability
Another benefit of automation is the early detection of defects. Tests that are part of an automated pipeline can detect defects at an earlier stage than manual or production tests. The earlier a defect is detected, the lower is the development cost, and the better is return on investment (ROI) for your automation.
Using Automation ROI analytics, you can also understand in what areas should you invest efforts to obtain the highest ROI, and how you can determine its effectiveness.
How automation ROI is measured
You can understand the ROI for test automation by examining the following automation objectives:
- Automate manual regression tests in order to save repetitive manual execution of these tests every sprint or release
- Release software versions faster and more frequently
- Detect defects earlier in the software development lifecycle
- Increase coverage and as a result reduce risk level and defects leakage
ALM Octane's widgets and insight graphs address these objectives to help you determine the ROI of your automation. The ROI trend analytics reflect the automation ROI by measuring the following metrics:
- Investment in automation vs. the release speed
- Regression savings (saved manpower days over manual testing)
- Defect cost
- Escaped defects
Calculate automation coverage per area
The widget configuration enables you to select a specific area in the product and define a filter for both manual and automated tests that you want to be included in this calculation.
To calculate the automation coverage, ALM Octane shows the automation coverage percentage increase/decrease for each release or sprint using the following formula:
The Automation ROI widget provides a graph with several metrics that can help you determine the ROI: development speed, regression cost, defect cost, and escaped defects.
These metrics are not intended to serve as a one-time score for a specific release or sprint, but rather reflect a trend of improvement, or ROI, by tracking the cost of defects over time.
The following table provides details about the graph metrics:
|Development cycle time||
Good automation should help increase the development speed. This manifests itself in several areas, such as:
The Automation ROI widget shows the trend of development speed by measuring the feature cycle time, and, if defined in the filter, per application module. For details on cycle time calculation, see Cycle and flow times.
By investing in automation, you can reduce repetitive execution of manual regression tests and improve your testing strategy by allowing QA to focus on other essential tasks.
The Regression Cost metric shows the number of manual runs done per release compared with automation coverage in this area for the current release.
By comparing the number of defects detected early by automation (low cost defects) vs. defects submitted after manual runs (high cost defects), you can determine the cost of defects by examining the following factors. This metric is calculated using ALM Octane analytics.
The Escaped defects metric reflects how many defects were missed by your testing activities. It also indicates whether your testing strategy is improving when you reduce the number of escaped defects that reach production.
You can also configure the way in which to track escaped defects, for example, by the environment on which the defect was detected.
Sample Automation ROI graph
The Automation ROI graph allows you to see trends that affect your ROI.
The x-axis is the selected time frame, such as release or sprint.
The y-axis represents the percentage change of automated tests in a release, compared to the first selected release/time frame. This allows you to see the investment vs. value since you began automating your tests.
The coverage bar indicates the number of automated tests out of the total number of tests.
In the following example, as the percentage of automated tests increased, the key indicators improved.
The percentages of the changes shown in the tooltip, compare the releases from the base point—the first release in which the key indicators had non-zero values.
For information about ROI insights and how to drill down further into the data, see Automation ROI insights.
The following metrics showed improvement in the later releases:
|Metric||Cause of improvement|
|Development cycle time||As the automation increased, there was an improvement in the development cycle time. Features were closed in less time since issues were detected at an early stage. Hover over a release to see the actual number of development days, as shown in this example.|
|Regression cost||As the automation increased, the regression cost decreased. This metric shows the number of manual runs (based on the manual tests filter) executed per release. Hover over a release to see the actual number of manual runs, as shown in this example.|
|Defect cost||As the automation increased, the defect cost decreased since defects were detected at an earlier stage.|
|Escaped defect||As the automation increased, there was a reduction in the number of escaped defects.|
Set the scope
Add the Automation ROI widget to your dashboard.
To customize the graph, click the vertical ellipsis and select Configure.
In the widget configuration area, define the following:
|General Scope||Indicate the application module, pipeline, and time frame, such as release and sprint, that you want to include in the calculation.|
Include regression tests that are candidates for automation.
Do not include acceptance tests, as they are not of primary relevance this calculation.
For example, you can apply the following filters:
To include all of the application module's descendants in the calculations, toggle the Descendants button:
- Include only direct descendants
- Include all descendants
Tip: The shortcut for switching between these options is Alt+i.
Configure the metrics
To customize and filter the metrics to be displayed b y the graph, click the vertical ellipsis and select Configure. Click Metrics:
Set the filter for defects that escaped being reported as defects. For example, you can filter this metric by users.
|Development Cycle Time||
Indicate the start and end phase for which to collect data. The defaults are In Progress and Done.
Filter the test runs. For example, you can filter automated runs coming from a specific pipeline job or package.