Testing overview

As you develop your application, testing is a critical part of the process. Testing ensures that the product meets your organization's quality standards.

Overview of test types

You can manage different types of testing processes.

Type	Details
Manual tests	In Manual tests, the tester uses the application to make sure that everything works as expected. Manual tests list each step and you perform the test, step by step. Gherkin tests are an advanced type of manual test, and contain scenarios that describe what to test and what to expect.
Automated tests	Automated tests are helpful for repetitive tasks and performing difficult tasks. Use automated tests for unit testing, regression testing, and continuous integration. You maintain automated tests in external testing tools. When working with automated tests, we recommend using the BDD testing framework. For details, see BDD testing framework.
Codeless tests	If you are working with the Functional Test module, the Design sub-module enables you to create codeless test automation scripts, using AI-based object identification. After you develop a codeless automation script in Design, you can associate it with a Codeless Test entity in the Quality module and include the script in your scheduled run. In addition, you can use Aviator's AI capabilities to generate automated codeless tests from your manual tests. You can also create codeless tests from the Tests tab and then open the scripts in Design. For details, see Functional Test Design.

Type

Details

Manual tests

In Manual tests, the tester uses the application to make sure that everything works as expected. Manual tests list each step and you perform the test, step by step.

Gherkin tests are an advanced type of manual test, and contain scenarios that describe what to test and what to expect.

Automated tests

Automated tests are helpful for repetitive tasks and performing difficult tasks. Use automated tests for unit testing, regression testing, and continuous integration. You maintain automated tests in external testing tools.

When working with automated tests, we recommend using the BDD testing framework. For details, see BDD testing framework.

Codeless tests

If you are working with the Functional Test module, the Design sub-module enables you to create codeless test automation scripts, using AI-based object identification.

After you develop a codeless automation script in Design, you can associate it with a Codeless Test entity in the Quality module and include the script in your scheduled run.

In addition, you can use Aviator's AI capabilities to generate automated codeless tests from your manual tests. You can also create codeless tests from the Tests tab and then open the scripts in Design.

For details, see Functional Test Design.

Test run results are incorporated into the dashboard and quality analysis.

Testing entities

The following table lists the various testing entities.

Test entity	Description
Manual Test	Manual tests make sure an application works as expected by testing most of its features. Manual test types include acceptance, end-to-end, regression, sanity, security, performance, and others. This entity represents the test steps to perform in the application under test. For details, see Manual testing.
Gherkin Test	Gherkin tests are a type of manual test. They use the Gherkin syntax and scenario structure. Gherkin test subtypes also include acceptance, end-to-end, regression, sanity, security, and performance. This entity represents the scenarios to perform on the application under test.
Automated Test	This entity represents the script that is stored and maintained in the external tool. Executable automated tests can run as part of a test suite. For details, see Automated testing and Add tests from an SCM repository. Note: For details on the Functional Test Execution integration, see Functional Test Execution.
Test Suite	A container with a set of tests grouped in one unit. Test suites contain no independent steps of their own. A test suite can contain: Manual and Gherkin tests that run sequentially. All manual and Gherkin tests run in the Manual Runner. Executable automation tests that are triggered to run on the CI server. When you run the test suite, the automated tests run on the CI server. You assign test runs to people, and run each test individually. After finishing all the test runs, the results of each test run are aggregated into a single suite run.
Manual Test Run	A manual run of a test. For usage examples, see Manual test syntax.
Gherkin Test Run	An automated run of a Gherkin test.
Test Suite Run	A run of a test suite.
Automated Test Run	A run of an automated test, by a CI server such as Jenkins or TeamCity.
Pipeline	Pipelines consist of pipeline steps, which represent the jobs that run on your CI server. Pipelines enable you to interact with your CI server. For example, pipelines collect the results of automated tests that run on the CI server.
Pipeline Run	A run of a pipeline, by a CI server such as Jenkins or TeamCity. Pipeline runs consist of builds, and can include information about failing automated tests, SCM commits related to this pipeline run, and more.
Build	A build is the result of a pipeline step run. A pipeline run consists of builds. If a pipeline step runs automated tests, its build consists of automated test runs.
Vulnerability	Vulnerabilities are security issues found in your code by a security testing scan. After reviewing a vulnerability, you can create a relevant defect to fix in your code, or dismiss and close the issue.

Test runs grid

The Runs tab in the Quality module provides a unified grid of runs across all tests, both planned and run. Testing managers can filter the grid, and do bulk updates on runs across different tests.

Note: By default, the Runs tab is accessible to workspace administrators only. For details on allowing access to other roles, see Roles and permissions.

Display limits

The Runs grid includes the first 2000 runs in the filter, according to the sort-by field. Refine the filter to make sure that all the runs you are interested in are included.

The following applies to grids with over 2000 runs:

You can sort the grid according to the following fields: ID, Started, Creation time, and Latest pipeline run.
If you sort by any other field, the grid does not display any results.
No results are displayed if the grid is grouped.

Test run statuses

Test runs have two status fields:

Native status. The test run's actual status. Can be any of the following values: Passed, Failed, Blocked, Skipped, In Progress, Planned
Status. A reduced list of test run statuses, used for widgets and coverage reporting. Can be any of the following values: Passed, Failed, Requires Attention, Skipped, Planned

The Status value is derived automatically from the Native status value. The two fields are mapped in the table.

"Native status" value	Mapped "Status" value
Passed	Passed
Failed	Failed
Blocked	Requires Attention
Skipped	Skipped
In Progress	Requires Attention
Planned	Planned