Create and run BDD scenarios

You can create and run BDD (behavior-driven development) tests in ALM Octane using Gherkin-syntax scenarios. Within ALM Octane you create specifications to describe the scenarios you want to test, manage the automation effort, and track run results.

In this topic:

Create and run scenarios

Use the Backlog module to add scenarios to BDD specifications.

To create and run scenarios:

1. Create, edit, and delete scenarios inside the feature context.

   When you create a script, ALM Octane generates tags for each scenario, so it can be later identified when results are injected from the CI.

   For details, see Create BDD specifications.

2. After you save the script, its scenarios are automatically populated in the BDD Scenarios tab, and you can run them manually. For details, see Run manual and Gherkin tests.

   Note: By default, the BDD Scenarios tab has a filter applied to show Is in latest version, so you only see scenarios that are currently relevant to your script. You can modify the filter as needed.

   If you remove a scenario from a script, its old run results are still saved. If you delete a scenario in the BDD Scenarios pane, its run results are also deleted, but the scenario is not deleted from the spec. The spec is an independent entity that can be edited as needed.

3. When a scenario is ready for automation, change its status in the Automation status field. If a spec is partially automated, this means that some of its scenarios are automated and some are manual.

4. You can now assign an owner who is responsible to automate the scenario.

   The owner sees the automation task in My Work, automates the test, the test runs in the CI, and results are pushed to ALM Octane to the scenario's runs.

   For details, see Automate Gherkin tests.

5. You can now drill down to the scenario's Runs tab and see results of its automated runs.

   Scenarios appear within the Tests modules in the same way as any other type of test.

Within a scenario's Details tab, you can click Go to BDD Spec and edit the script where the scenario is defined.

If you changed the spec and now are not aligned with the code in the SCM repository, the spec's Code alignment field shows that you need to align between ALM Octane and the SCM.

You can accept or reject the external code to determine which option is active. For details, see Prepare Gherkin tests for automation.

Back to top

Automated scenario injection

You can inject your .feature file via the CI to ALM Octane, and automatically create specs and scenarios.

Follow the flow described in Automate Gherkin tests, but note the following mandatory steps to enable the use of a BDD spec model.

Mandatory steps:

1. After you set up a pipeline in ALM Octane, open the pipeline's Topology tab and access the configuration of each relevant job.

2. By default, new content is injected as a BDD specification, as defined by the BDD_INJECT_SPECS_TYPE_BY_DEFAULT space parameter. This means that a separate run is created for each scenario in the injected new feature file. You can override the parameter setting by selecting Inject new BDD content as Gherkin test/BDD Specification in a pipeline step.

   In spaces where pipelines were already created: By default, each job is set to Inject new BDD content as Gherkin test. This means that new tests are injected so that one run is created for all the scenarios in a script. To work with BDD granularity as described above, select Inject new BDD content as a BDD Specification. A separate run is created for each scenario in the injected new feature file. This setting impacts all of the jobs that are nested below the current job.

3. Within the feature file, each scenario must have a unique name. If you try to inject a scenario that does not have a name or if you have two scenarios with the same name, ALM Octane alerts you to the problem.

Back to top

Automation status on the BDD scenario

Automation status is defined on BDD scenarios as follows:

• Scenarios that are created in ALM Octane start as Not Automated. They can be manually set as Ready for automation. Once ALM Octane receives an automated run for a scenario, it becomes Automated.

• Scenarios that are created as a result of run injection are created as Automated from the start.

There is also an automation status field on the BDD spec, which serves as the summary of its scenarios.

Back to top

Inject BDD test results to ALM Octane

The bdd2octane utility enables you to inject test results from all common BDD frameworks to ALM Octane, via your CI.

You provide the tool with a path to your feature files, and a path to the report files produced by your BDD execution tool.

The tool takes the information from the JUnit XML report generated by your BDD framework, together with the feature file in Gherkin syntax, and sends the result to ALM Octane using the Application Automation Tools plugin. The tool is configured as a build step before ALM Octane Cucumber test reporter, and after the step that runs the tests.

You can inject BDD test results via an executable .jar file, or using a Maven plugin. This solution is open-sourced, to enable you to add custom frameworks. For details, refer to the bdd2octane tool.

The utility is extensible to any BDD framework using a JUnit report.

Back to top

Inject BDD test results from SpecFlow

You can inject BDD test results from SpecFlow, using a report generation template file. For details, see ALM Octane BDD Automation with SpecFlow.

In this process, you add a .cshtml template file to your test solution. This template is then used by SpecFlow when it generates the text report xml.

The generated reports are injected to ALM Octane as Gherkin automated runs. The first run of the pipeline generates a Gherkin automated run entity that is linked to the existing Gherkin test. If the Gherkin test does not exist, it is created in ALM Octane. The following runs of the automated test are generated under the Previous Runs tab of the initial Gherkin automated run.

In the Pipeline section, the automated test results can be seen inside each run of the build in the Tests tab.

Back to top

How test results are counted

Test results are counted differently in Jenkins and in ALM Octane.

For example, suppose you have a Gherkin feature file run by Jenkins and injected to ALM Octane as a BDD spec. This feature file has one normal scenario, another outline scenario with two examples, and one scenario tagged as excluded.

When you look at the run result in Jenkins, it shows three test runs, because the normal scenario is counted once, the outline scenario (with two examples) is counted twice, and the excluded scenario does not run and therefore is not counted.

When the same feature file is injected to ALM Octane as a BDD specification, ALM Octane counts two tests: the normal scenario is counted once, and the outline scenario with two examples is counted once. ALM Octane treats a scenario outline as a single BDD scenario entity, no matter how many iterations it has.

Back to top

Generate example-level reports

You can set up Dashboard widgets to report on runs of examples in a scenario outline. This means that each example is counted separately as a run. This is available for the following widgets:

• Summary graphs: Test run history (manual), Last test runs, Backlog's last test runs

• Traceability graph: Backlog's last test runs, Test’s last test runs

To enable example-level reporting, when configuring a supported widget select the checkbox Count iterations as separate runs.

The status of an iteration is updated automatically based on the status you assign to the steps in that iteration when running the test. If you do not assign a status to the steps, the iteration status will remain Planned.

Version compatibility

• Runs created in ALM Octane versions earlier than 16.1.200 have an iteration count of 1.

• Planned runs created in ALM Octane versions earlier than 23.4 have an iteration count of 1.

Next steps:

• Create BDD specifications
• Run manual and Gherkin tests
• Automate Gherkin tests