Automate Gherkin tests

Create automated tests from your Gherkin tests, add them to an automation project, and then run them in your CI server builds.

Gherkin automation process

The following components are involved in automating Gherkin tests.

Component	Process
OpenText Core Software Delivery Platform	Create Gherkin tests Download test scripts to be uploaded to the IDE View automated test run results and analyze them
Your IDE	Add automation for the test, through a .feature file included in a JUnit Runner automation project.
CI system	Run the test as a part of a build or pipeline run

There are several different flows, depending on the stage of development and your role.

Stage and role	Flow
Admin	The following is a one-time flow for admins: Install the OpenText Core SDP CI plugin. Create a build job that reports Gherkin test results. Create a pipeline.
Developer	Configure the automation project (octane-cucmber-jvm library and other settings).
Single Test Automation	The following is a single test automation flow that can be configured for each test run: Download the test using OpenText Core SDP or IDE plugin. Add the automation code. Commit changes. Perform pipeline runs in the CI. Results are imported to OpenText Core SDP. See results - status becomes Automated.

Stage and role

Flow

Admin

The following is a one-time flow for admins:

Install the OpenText Core SDP CI plugin.
Create a build job that reports Gherkin test results.
Create a pipeline.

Developer

Configure the automation project (octane-cucmber-jvm library and other settings).

Single Test Automation

The following is a single test automation flow that can be configured for each test run:

Download the test using OpenText Core SDP or IDE plugin.
Add the automation code.
Commit changes.
Perform pipeline runs in the CI.
Results are imported to OpenText Core SDP.
See results - status becomes Automated.

Prerequisites

Make sure that the following prerequisites are met:

The CI plugin is installed and configured on the CI server. For details, see CI server integration.
Your automation project is set up as a Cucumber project.

Create and download the Gherkin test

To work with a Gherkin test in your IDE, you must create the test.

To create and download the test:

If necessary, create and add scenarios to your Gherkin test. For details, see Create Gherkin tests.
In the test's Script tab, click the Download script button .

The script is downloaded as a .feature file.

Note: Do not delete the Test ID tag; it is used to map the automation results.
Add the .feature file to your automation project.

Configure your project

Note: You can use the following procedure to enable BDD test result injection in versions earlier than 16.0.200. This method is limited to Cucumber-jvm up to version 4.8.1, and JBehave.

Beginning with version 16.0.200, there is a more advanced method that can be used to inject test results by all major BDD frameworks, which does not require any change to your code. For details, see Inject BDD test results.

Depending on your integration, configure the following in your project.

Integration	Description
Cucumber	To generate a Cucumber-based test results file that the CI server sends to OpenText Core SDP, add the octane-cucumber-jvm library to your automation project (supported for Java environments only). For configuration details, see octane-cucumber-jvm.
JBehave	Configure the octane-jbehave-gherkin-reporter in your project, as described in octane-jbehave-gherkin-reporter. When running the tests, the reporter outputs files with the results. The CI plugin reads these files and uploads the results back to OpenText Core SDP.
Azure DevOps Server	To enable injection of Gherkin test results from Azure DevOps Server, configure the octane-azure-devops-extension as described in octane-azure-devops-extension. For details, see Displaying Cucumber Gherkin test results in HowToInstall.docx.

Create a build job in your CI

In your build job configuration, add a build step that runs the test.

If required by your CI, add a post-build step to copy the test results to the job folder. For example, in Jenkins, add an OpenText Core Software Delivery Platform Cucumber test reporter step from the actions section.

Note:

If you add a Cucumber test reporter step, the CI server sends only the Gherkin test results. Test results from other automated tests are not sent as part of the build.
The CI Server must be configured with sufficient memory to inject large Cucumber test results. For example, for Jenkins, the minimum recommended memory is 2 GB.

The post-build action instructs the CI to copy the test results XML files from the path specified in the post-build actions to the job build folder. The results are copied from this build folder to OpenText Core SDP.

During the test run in the CI server, the OctaneCucumber runner generates a run results report called <test name>_OctaneGherkinResults.xml in a dedicated folder called gherkin_results.

Note: If you are using the octane-cucumber-jvm library version 12.53.19 and earlier, the file name of the results is called OctaneGherkinResults.xml.

If the post-build Cucumber test report step fails, the CI server fails the entire build.

Add the build job to a pipeline

Ensure that your build project or build job that runs the test is included in a pipeline. When you run the pipeline, your CI generates the test results. The CI plugin reads the results file and sends the details to OpenText Core SDP.

For details on creating and working with pipelines, see Create and configure pipelines.

Run the pipeline

To send the results to OpenText Core SDP, you must run the corresponding pipeline.

To run the pipeline:

In OpenText Core SDP or in the CI server, run the pipeline.

The test results are linked to the original Gherkin tests using the Test ID tags found at the top of the .feature file and the test script. The test is assigned using these criteria:
- If the test already exists in OpenText Core SDP and the Test ID tag is included in the .feature file, the mapping is done by the Test ID tag.
- If the Test ID tag is not included in the .feature file, the mapping is done by the .feature script path.
- If the test does not exist in OpenText Core SDP, a new test is created and the results are mapped to it.
Analyze the results in OpenText Core SDP.

Unique Gherkin tests and test runs

Unique automated Gherkin tests and test runs are identified in the following manner.

Entity Description

Unique Gherkin tests

Entity	Description
Unique Gherkin tests	Each Gherkin test is identified by its script feature's tag, if one exists. The tag exists if the Gherkin test was downloaded from OpenText Core SDP at some point. The script has a tag that contains the test ID and revision in the format: `@TID4002REV0.2.0`. When a Gherkin test run is received, its attributes are checked to associate the run with a specific Gherkin test. If no existing Gherkin test matches the run, a new automated Gherkin test entity is created.
Unique Gherkin test runs	Each Gherkin test can have more than one run, each uniquely identified by test (which was uniquely identified by its tag or path), pipeline, suite run id, and external run id. This is in addition to the fields that define a Last Run: Release, Milestone, Program, and Environment. For details, see Last runs. The details of the Gherkin automated run are stored for each unique combination of these attributes. Each run accumulates its own history of Previous Runs. For example, if a Gherkin test was run on Chrome and Firefox in two different pipelines, the Gherkin test would have four Gherkin automated runs. Two of these runs would be considered last runs.

Each Gherkin test is identified by its script feature's tag, if one exists.

The tag exists if the Gherkin test was downloaded from OpenText Core SDP at some point. The script has a tag that contains the test ID and revision in the format: @TID4002REV0.2.0.

When a Gherkin test run is received, its attributes are checked to associate the run with a specific Gherkin test. If no existing Gherkin test matches the run, a new automated Gherkin test entity is created.

Unique Gherkin test runs

Each Gherkin test can have more than one run, each uniquely identified by test (which was uniquely identified by its tag or path), pipeline, suite run id, and external run id. This is in addition to the fields that define a Last Run: Release, Milestone, Program, and Environment. For details, see Last runs.

The details of the Gherkin automated run are stored for each unique combination of these attributes. Each run accumulates its own history of Previous Runs.

For example, if a Gherkin test was run on Chrome and Firefox in two different pipelines, the Gherkin test would have four Gherkin automated runs. Two of these runs would be considered last runs.

Troubleshooting

This section contains troubleshooting suggestions if you are unable to see the automated test that was run.

Checks	Description
Check your code	Make sure that you are using the correct syntax: Cucumber does not always indicate syntax errors. Run your test locally.
Check that an XML was generated	If you were able to run the test locally, check that an XML was generated. Check that your POM file is defined properly and includes the octane-cucumber-jvm library along with the correct version, and make sure that OctaneCucumber is defined as the runner in your test class. For details, see Configure your project. If both POM file and test class have been updated properly and an XML is not generated, contact your support representative.
Check test ran in your CI tool	If the test runs locally, check that the test ran in your CI tool. Make sure the CI plugin is installed and configured on the CI server. For details, see CI server integration.
Check the job's console log	Check the job's console log to make sure that the post-build action found the XML: `OpenText Core Software Delivery Platform Cucumber test reporter: Got result file content` `OpenText Core Software Delivery Platform Cucumber test reporter: Result file copied to …`
Check the plugin logs	The plugin logs should display: `Successfully pushed test results of build [<build name>#<build number>]`
Check the server logs	Check the server logs for errors. XSD validation errors are common. For example, if you ran the test x times, but instead of seeing x runs under a single test, you see X tests with one run each.
Check the test script in the .feature file	Check that the test script in the .feature file includes the test ID (TID). The test ID is required for mapping automation results. If the TID is missing, download the test script and implement it again.