Create and run BDD scenarios
You can create and run BDD (behavior-driven development) tests in ALM Octane using Gherkin scenarios, with the Cucumber-JVM or JBehave frameworks. Within ALM Octane you create specifications to describe the scenarios you want to test, manage the automation effort, and track run results.
The following is an example of working with BDD tests using ALM Octane and Gherkin.
A business persona creates a feature in ALM Octane.
Together with the testing team, they create acceptance tests for the feature in Gherkin.
These tests can be run manually in ALM Octane, or can be marked as ready for automation.
When the developer sees which tests need automation, they take the contents of the feature file, namely of the Gherkin test or BDD spec, and save it locally in the IDE. They then automate the test and push the code using Cucumber-JVM or JBehave.
The automated tests are now run in the CI and results are reported to ALM Octane.
You can now see test results and coverage data within ALM Octane, in the context of the original feature.
Another common flow is when a tester creates a feature file in the code, and the test is injected to ALM Octane. This test can then be assigned to a backlog item or application module in ALM Octane.
Use the Backlog module to add BDD specifications and their associated scenarios. When you create a script, ALM Octane generates tags for each scenario, so it can be later identified when results are injected from the CI.
For details, see Create BDD Specifications.
After you save the script, its scenarios are automatically populated in the BDD Scenarios tab, and you can run them manually. For details, see Run manual and Gherkin tests.
Note: By default, the BDD Scenarios tab has a filter applied to show Is in latest version, so you only see scenarios that are currently relevant to your script. You can modify the filter as needed.
If you remove a scenario from a script, its old run results are still saved. If you delete a scenario in the BDD Scenarios pane, its run results are also deleted, but the scenario is not deleted from the spec. The spec is an independent entity that can be edited as needed.
When a scenario is ready for automation, change its status in the Automation status field. If a spec is partially automated, this means that some of its scenarios are automated and some are manual.
You can now assign an owner who is responsible to automate the scenario, and the workflow now is the same as when automating a Gherkin test.
The owner sees the automation task in My Work, automates the test, the test runs in the CI, and results are pushed to ALM Octane to the scenario's runs.
For details, see Automate Gherkin tests.
You can now drill down to the scenario's Runs tab and see results of its automated runs, as with any Gherkin test.
Scenarios appear within the Tests modules in the same way as any other type of test.
Within a scenario's Details tab, you can click Go to BDD Spec and edit the script where the scenario is defined.
If you change the spec and are now not aligned with the code in the SCM repository, the spec's Code alignment field shows that you need to align between ALM Octane and the SCM.
You can accept or reject the external code to determine which will be active. For details, see Prepare your Gherkin tests for automation.
In the Dashboard, you can create widgets at the spec or scenario level. For example, you can select Last test runs as item type and BDD Spec as an additional filter, then group by spec to see the last scenario run status per spec.
Automated scenario injection
Similar to the Gherkin test injection flow, you can inject your .feature file via the CI to ALM Octane. and automatically create specs and scenarios.
Follow the flow described in Automate Gherkin tests, but note the following mandatory steps to enable use of the BDD spec model:
After you set up a pipeline in ALM Octane, open the pipeline's Topology tab and access the configuration of each relevant job.
By default, each job is set to Inject new BDD content as Gherkin test. This means that new tests are injected so that one run is created for all the scenarios in a script.
To work with BDD granularity as described above, select Inject new BDD content as BDD Specification. A separate run will be created for each scenario in the injected new feature file. This setting will impact all of the jobs that are nested below the current job.
Within the .feature file, each scenario must have a unique name. If you try to inject a scenario that does not have a name or if you have two scenarios with the same name, ALM Octane will alert you to the problem.
- We recommend that you follow the pipeline which injects scenarios, in order to get notifications if the results injection fails.
- Do not use the same CI job to inject BDD content in two different workspaces. This may result in inconsistent BDD run results.
- In addition, do not inject the same .feature file using different jobs if the jobs have a different configuration (inject as Gherkin vs. BDD specs).
Understanding how test results are counted
Test results are counted differently in Jenkins and in ALM Octane.
For example, suppose you have a Gherkin feature file run by Jenkins and injected to ALM Octane as a Gherkin test. This Gherkin feature file has one normal scenario, another outline scenario with two examples, and one scenario tagged as excluded.
When you look at the run result in Jenkins it shows three test runs, because the normal scenario is counted once, the outline scenario (with two examples) is counted twice, and the excluded scenario does not run and therefore is not counted. In ALM Octane it counts as one test only, because the entire feature file is counted once.
If the same feature file is injected to ALM Octane as a BDD specification, ALM Octane counts two tests: the normal scenario is counted once, and the outline scenario with two examples is counted once. ALM Octane treats a scenario outline as a single BDD scenario entity, no matter how many iterations it has.