Create and run BDD scenarios
You can create and run BDD (behavior-driven development) tests in ALM Octane using Gherkin scenarios. Within ALM Octane you create specifications to describe the scenarios you want to test, manage the automation effort, and track run results.
The following is an example of working with BDD tests using ALM Octane and Gherkin.
A business persona creates a feature in ALM Octane.
Together with the testing team, they create acceptance tests for the feature in Gherkin.
These tests can be run manually in ALM Octane, or can be marked as ready for automation.
When the developer sees which tests need automation, they take the contents of the feature file, namely of the Gherkin test or BDD spec, and save it locally in the IDE. They then automate the test and push the code to ALM Octane.
The automated tests are now run in the CI and results are reported to ALM Octane.
You can now see test results and coverage data within ALM Octane, in the context of the original feature.
Another common flow is when a tester creates a feature file in the code, and the test is injected to ALM Octane. This test can then be assigned to a backlog item or application module in ALM Octane.
Available in versions: 16.0.200 and later
Use the Backlog module to add BDD specifications and their associated scenarios. You can create, edit, and delete scenarios inside the feature context.
When you create a script, ALM Octane generates tags for each scenario, so it can be later identified when results are injected from the CI.
For details, see Create BDD Specifications.
After you save the script, its scenarios are automatically populated in the BDD Scenarios tab, and you can run them manually. For details, see Plan and run manual and Gherkin tests.
Note: By default, the BDD Scenarios tab has a filter applied to show Is in latest version, so you only see scenarios that are currently relevant to your script. You can modify the filter as needed.
If you remove a scenario from a script, its old run results are still saved. If you delete a scenario in the BDD Scenarios pane, its run results are also deleted, but the scenario is not deleted from the spec. The spec is an independent entity that can be edited as needed.
When a scenario is ready for automation, change its status in the Automation status field. If a spec is partially automated, this means that some of its scenarios are automated and some are manual.
You can now assign an owner who is responsible to automate the scenario. The workflow is the same as when automating a Gherkin test.
The owner sees the automation task in My Work, automates the test, the test runs in the CI, and results are pushed to ALM Octane to the scenario's runs.
For details, see Automate Gherkin tests.
You can now drill down to the scenario's Runs tab and see results of its automated runs, as with any Gherkin test.
Scenarios appear within the Tests modules in the same way as any other type of test.
Within a scenario's Details tab, you can click Go to BDD Spec and edit the script where the scenario is defined.
If you change the spec and are now not aligned with the code in the SCM repository, the spec's Code alignment field shows that you need to align between ALM Octane and the SCM.
You can accept or reject the external code to determine which will be active. For details, see Prepare your Gherkin tests for automation.
In the Dashboard, you can create widgets at the spec or scenario level. For example, you can select Last test runs as item type and BDD Spec as an additional filter, then group by spec to see the last scenario run status per spec.
If your scenario includes tags, they are displayed in ALM Octane in the Gherkin tags field, both for BDD specs (at the feature file level) and BDD scenarios (at the feature and scenario level). You can then perform actions such as filter by tags or use them in reporting.
Automated scenario injection
Similar to the Gherkin test injection flow, you can inject your .feature file via the CI to ALM Octane, and automatically create specs and scenarios.
Follow the flow described in Automate Gherkin tests, but note the following mandatory steps to enable use of the BDD spec model:
After you set up a pipeline in ALM Octane, open the pipeline's Topology tab and access the configuration of each relevant job.
By default, each job is set to Inject new BDD content as Gherkin test. This means that new tests are injected so that one run is created for all the scenarios in a script.
To work with BDD granularity as described above, select Inject new BDD content as a BDD Specification. A separate run will be created for each scenario in the injected new feature file. This setting will impact all of the jobs that are nested below the current job.
Within the feature file, each scenario must have a unique name. If you try to inject a scenario that does not have a name or if you have two scenarios with the same name, ALM Octane will alert you to the problem.
- We recommend that you follow the pipeline which injects scenarios, in order to get notifications if the results injection fails.
- Do not use the same CI job to inject BDD content in two different workspaces. This may result in inconsistent BDD run results.
- In addition, do not inject the same .feature file using different jobs if the jobs have a different configuration (inject as Gherkin vs. BDD specs).
Available in versions: 16.0.200 and later
The bdd2octane utility enables you to inject test results from all common BDD frameworks to ALM Octane, via your CI.
You provide the tool with a path to your feature files, and a path to the report files produced by your BDD execution tool.
The tool takes the information from the JUnit XML report generated by your BDD framework, together with the feature file in Gherkin syntax, and sends the result to ALM Octane using the Micro Focus Application Automation Tools plugin. The tool is configured as a build step before ALM Octane Cucumber test reporter, and after the step that runs the tests.
You can inject BDD test results via an executable .jar file, or using a Maven plugin. This solution is open-sourced, to enable you to add custom frameworks.For details, see https://github.com/MicroFocus/bdd2octane.
This utility is extensible to any BDD framework using a JUnit report.
Available in: ValueEdge and ALM Octane 16.0.400 and later
You can inject BDD test results from SpecFlow, using a report generation template file. For details, see ALM Octane BDD Automation with SpecFlow.
In this process, you add a .cshtml template file to your test solution. This template is then used by SpecFlow when it generates the text report xml.
The generated reports are injected to ALM Octane as Gherkin automated runs. The first run of the pipeline generates a Gherkin automated run entity that is linked to the existing Gherkin test. If the Gherkin test does not exist, it is created in ALM Octane. The following runs of the automated test are generated under the Previous Runs tab of the initial Gherkin automated run.
In the Pipeline section, the automated test results can be seen inside each run of the build in the Tests tab.
Understanding how test results are counted
Test results are counted differently in Jenkins and in ALM Octane.
For example, suppose you have a Gherkin feature file run by Jenkins and injected to ALM Octane as a Gherkin test. This Gherkin feature file has one normal scenario, another outline scenario with two examples, and one scenario tagged as excluded.
When you look at the run result in Jenkins it shows three test runs, because the normal scenario is counted once, the outline scenario (with two examples) is counted twice, and the excluded scenario does not run and therefore is not counted. In ALM Octane it counts as one test only, because the entire feature file is counted once.
If the same feature file is injected to ALM Octane as a BDD specification, ALM Octane counts two tests: the normal scenario is counted once, and the outline scenario with two examples is counted once. ALM Octane treats a scenario outline as a single BDD scenario entity, no matter how many iterations it has.