Test suites
A test suite is a collection of tests. Use test suites to test an area of your application. You can use test suites for planned testing events, such as regression testing.
A test suite can contain manual, Gherkin, and automated tests. For details, see Testing overview.
About test suites
A test suite is a collection of tests. Test suites do not contain any content of their own.
Use test suites in a variety of ways:
- Group tests of a particular page, module, or section of your application together. This lets you test that section of the application.
- Keep tests of a particular theme grouped together. You can reuse the same test suite across versions, and compare the suite run from the current version with those from previous versions. In addition, you can run the same suite on different environments and compare suite runs to check for any regressions.
- Enforce the order of a series of tests. You can ensure that a series of manual tests are run in a specific order, so that a test cannot be run until the previous tests in the test suite have been completed.
- Different people run the tests. The owner of the test suite receives the compiled results for the test suite's test. This method is particularly helpful for regression testing.
Tip: The best practice is to create maintainable and reusable test suites with a relatively small number of tests. You can organize your tests in suites based on one of the suggestions above, and then analyze the results across suites at a higher level, such as release, milestone, or environment.
Create a test suite
Create a test suite to contain the tests you want to group together.
To create a test suite:
-
In the Quality or Backlog module, open the Tests page.
- If you want to add tests directly to the new test suite, select those tests in the grid.
- Right-click in the tests grid, and select Create Test Suite.
-
Set the test suite attributes. In particular, provide a value for the Backlog coverage and Application modules attributes.
-
Click Add. The test suite is created.
Tip: To add tests to the new test suite immediately, click Add & Edit. For details on adding tests, see Add tests to a test suite.
Add tests to a test suite
You can add tests to a test suite either from within the test suite, or from the Tests page.
You can add the same test to a suite multiple times, so for example you could run an automated test twice using different environments or execution parameters.
To add tests to a test suite from within the test suite:
-
In the Quality or Backlog module, open the Tests page.
- Open a test suite and select its Planning tab.
-
To add existing tests to the test suite:
- Click the Add Existing Tests button
.
- Select all the tests you want to include.
- Click Add.
- Click the Add Existing Tests button
-
To add new tests to the test suite:
- Click + Manual Test or use the down arrow to select Gherkin Test.
- Set values for the test fields. For details, see Create manual tests or Gherkin tests.
- Click Add. The new test is created and automatically added to the test suite.
To add tests to a test suite from the Tests page:
-
In the Quality or Backlog module, open the Tests page.
-
Select the tests you want to add to a test suite.
-
Right-click and select Add to Test Suite.
-
In the Add to Test Suite dialog box, select all the test suites to which you want to add the tests, and click Add.
Note: After you add tests to a suite, any changes you make to the Backlog coverage, Test runner, or Data set field apply only within that particular test suite. The changes do not affect the test outside of the test suite.
Parameterize tests
You can parameterize your test runs using data tables.
Test type | Description |
---|---|
Manual |
If you add manual tests to a test suite, you can parameterize them as described in Use parameters in tests. |
Automated |
If you add executable OpenText Functional Testing tests to a test suite and your OpenText Functional Testing Git repository contains data tables, parameterize your test runs using data tables. For details, see Add tests from an SCM repository. Tip: For the data tables to be located in Git, store them in an entirely separate folder from your tests. To parameterize your test runs using data tables:
When the suite runs this test, it runs with the data table you selected. To run a test with different data tables, add the test to different test suites. In each suite, you can specify a different data table for this test. |
Plan a test suite run
After you create a test suite, prepare the test suite to run. Assign owners and environment tags to each of the tests.
The test suite run process follows these steps:
Tip: To apply the same plan to multiple test suites, in the tests grid, select the necessary test suites and click the Plan Run button .
To plan a test suite run:
- On the Tests tab, open a test suite.
-
On the test suite's Planning tab, manage the tests you want to run.
Action Steps Add new or existing tests
Use the toolbar buttons to add new or existing tests.
Tip: You can add the same test multiple times if, for example, you want to run the same test with different environment tags.
Remove unwanted tests Select the test's checkbox and select More > Remove from test suite.
Select tests to include in the next run Indicate which tests you want to include in the next run by turning on or off the switch in the Include in next run column. By default, all tests are set to run.
Note: The checkboxes next to the test ID do not indicate whether to run the test. Selecting these checkboxes applies to other features, such as exporting, deleting, and reporting.
Set the test run order Drag and drop the tests to set the order in which they should be run.
To send a test to the top or bottom of the list, right-click the test and select Rank Highest or Rank Lowest.
The test run order indicates the priority of the tests in the suite, or the recommended order in which to run the tests. By default, the run order is not enforced. You can configure the suite to enforce the run order for manual tests. For details, see Enforce manual test run order.
Add a test to a planned suite run To add a test to a planned suite run, right-click the test and select Add to suite run. In the Add to Suite Run dialog box, select the suite runs to which you want to add the test, and click Add & Close.
You can also add a run that already exists. This option is available for both manual and automated tests.
-
In the Planning tab, click the Plan Suite Run button
. The Plan Suite Run dialog box lets you define the following suite's attributes.
Field Details Release Select a release that you want to test. After you select a release, you can select a milestone or sprint, but they have different functions:
- Milestone: When you select a milestone, it means that the run contributes to the quality status of the milestone. A unique last run is created and assigned to the milestone for coverage and tracking purposes, similar to release.
- Sprint: When you select a sprint, it means that the run is planned to be performed in the time frame of the sprint. Sprints do not have their own unique run results, so filtering Last Runs by Sprint returns only runs that were not overridden in a later sprint. To see full run status and progress reports based on sprints, use the Test run history (manual) item type instead of using Last Runs in the Dashboard widgets.
These values are assigned to all test runs for the tests in the suite.
Tip: The values of user-defined fields are also assigned, if the user-defined fields have the same name and type in both the test suite run entity and its test run entities. This is available if the COPY_RUN_SUITE_UDFS_TO_RUNS configuration parameter is set to true.
Program (Optional) If you are working with programs, you can select a program to associate it with the suite. All of the runs in the suite are assigned to this program, which is recorded in each run’s history. For details, see Programs. Default environment tags and Default run by Specify the Default environment tags and Default run by values to be used by all test suite runs.
The test suite run's Environment tags and Run by fields inherit the default values. If the Default run by field is left empty, test runs are not assigned to any user. You need to manually assign each of the planned test runs to a user.
Data access: If a test belongs to a data category that falls outside of the default tester's role, the run is not assigned to the default user. Manually assign the test run to an accredited user. For details, see Data access control.
Note: If you modify either of these fields after planning a test suite, the changes only affect new runs. The existing runs continue to inherit the original default values.
Enforce order for manual When running a suite containing manual tests, set this field to Yes to enforce the run order of the tests in the suite. For details, see Enforce manual test run order.
You can change the value of this field for a suite run only when creating the suite run. After a suite run has been created, this field becomes read-only and cannot be changed.
Tip: This field might not be visible by default. To show it, click the Customize Fields button
and select Enforce order for manual from the list.
-
In the Plan Suite Run box, click Plan.
The suite run and test runs for the suite's tests are created or updated.
-
Users can run the planned tests from one of the following areas.
Area | Details |
---|---|
My Work module |
Test runs are displayed in the user's item list in the My Work module. Each assigned run is labeled with the name of the suite run and a link to the individual test.
Select the test and click the Run button If all tests in the suite are assigned to the same user, they appear in My Work as a suite run. Select the suite run and click the Run button |
Team Backlog or Quality modules |
On the Details tab of the suite run, click the Run/Run Suite button If you are running the entire test suite, and some tests are not assigned, indicate whether to run your assigned tests with the unassigned tests or all the tests in the test suite. |
-
(Optional) You can export the test suite information in the Excel or CSV format.
The export saves all fields, with the exception of UDFs (user-defined fields), and the following built-in fields: Author, Backlog coverage, BDD Spec, Builds, Component, Covered automated test, Covered manual test, Covered requirement, Data table, External test ID, Flag rules, Flags, Followed by me, In my work, Is draft, Is in latest version, Manual, Owner, Quality stories, and Test level.
Note: When you plan a suite run, it may take some time for you to see the planned runs, since they are created in an asynchronous manner.
Run a test suite
A test suite runs as a collection of test runs.
By default, all the tests included in the test suite are run unless certain tests were excluded in the planning stage.
Note: When a test suite runs, the included test runs are created in draft mode. The Alert User business rules that govern test runs creation are ignored.
You can prevent creating test runs in draft mode by setting the IGNORE_ALERT_RULE_WHEN_CREATE_IN_DRAFT_MODE parameter to false. As a result, the Alert User business rules are triggered, and the runs cannot be created as drafts. When a test suite runs, an error occurs.
For details, see Configuration parameters.
To run a test suite:
-
On the test suite Planning tab, verify that the necessary tests are included in the run.
To change the tests included in the run, in the Include in next run column, select the tests to run. Note that checkboxes next to the test ID do not mark the tests to run.
- Click the Run Suite button
. The Run <test suite name> dialog box opens.
-
Verify the settings and click Let's Run.
The Manual Runner window opens. For details, see Run manual and Gherkin tests.
The status of the test suite is automatically updated according to the status of the test runs.
Note: The final status of the test suite follows the business rule logic. The business rules run sequentially and the final status is dictated by the last rule. For example, the first built-in business rule sets the test suite status to Passed, if a test suite's tests were either Skipped or Passed. However, a subsequent rule whose criteria is to look for any Skipped tests, resets the status of the test suite to Skipped. The business rules and their order are set by the admin. For details, see Business rules.
Manage suite runs
The suite run's Runs tab lets you manage the runs by changing their rank or other attributes. Use the right-click menu to add items to My Work or to send them as an email.
Tip: To update details for multiple test runs, do a multiple selection and choose Bulk Update from the context menu.
To manage the suite runs:
-
Select the Suite Runs tab. A list of all suite runs is displayed.
-
Click a suite run. The Runs tab opens and displays a list of runs.
-
To change the status of a run, modify the Native status in the Details tab and click Save. For example, if a run was previously blocked, you can change its status to Planned.
-
If you want to run a test twice as part of a suite run, make a copy of the run by selecting it and clicking Duplicate Run. If you want to redo a run from scratch, duplicate the run, make sure the copy's status is set to Planned (this is the default), and delete the old one.
-
Rank and reorder the runs by dragging them to the desired location.
You cannot change the order of a run which is currently running.
Tip: To ensure that the results stay aligned with their corresponding runs, keep the runs in the same order as they appear in the suite run, and do not run tests in parallel.
-
If you need to change the test assignment or an environment tag, modify the values in the Run by and Environment tag columns. The assignees receive a notification in their My Work area.
Note: The Environment tag is an information label. The environment on which the test runs is not selected or detected.
-
After you assign all the tests, the Runs tab displays the updated list.
Enforce manual test run order
When planning a run of a test suite containing manual tests, you can configure the suite to enforce the run order of the manual tests in the suite. This means that a manual test run cannot be run until the preceding manual test runs in the suite have been completed.
Note: Enforcing test run order is not supported for automated tests. In a suite containing both automated and manual tests, the automated tests can be run at any time, and their run status is ignored when enforcing the run order of manual tests in the suite.
To enforce test run order for a suite run, do one of the following:
-
Before creating the suite run, select the Details tab of the test suite and set the Enforce order for manual field to Yes.
The value you set at the test suite level is the default for all future suite runs planned for the suite.
-
When creating the suite run, set the Enforce order for manual field for the suite run to Yes. For details, see Plan a test suite run.
Note: After a suite run has been created, the Enforce order for manual field becomes read-only and cannot be changed.
Test run completion status
By default, a test run is considered completed when its status is Passed, Failed, or Skipped. The administrator can choose which statuses are considered completed, by modifying the relevant business rule for the Manual Run entity. For details, see Business rules.
Tip: The administrator can define a business rule to automatically notify the assigned tester when the next test is ready to run. For details, see Notify the assigned tester when a test is ready to run.
If a test run in the suite was completed, and its run status was then changed back to Planned after subsequent tests were already allowed to run, the updated status affects all subsequent test runs. This means that all test runs after the run marked Planned cannot be run.
Example: A suite contains 5 manual test runs. Test runs 1, 2, and 3 passed. Test run 4 can currently be run.
Run order | Status | Blocked by previous run | Can be run |
---|---|---|---|
Test run 1 | Passed | No | Yes |
Test run 2 | Passed | No | Yes |
Test run 3 | Passed | No | Yes |
Test run 4 | Planned | No | Yes |
Test run 5 | Planned | Yes | No |
Then, Test run 2 is updated from Passed to Planned. Now, Test runs 3 and 4 cannot be run.
Run order | Status | Blocked by previous run | Can be run |
---|---|---|---|
Test run 1 | Passed | No | Yes |
Test run 2 | Planned | No | Yes |
Test run 3 | Passed | Yes | No |
Test run 4 | Planned | Yes | No |
Test run 5 | Planned | Yes | No |
To override this restriction and allow subsequent tests to run, you can manually change the Is completed field to Yes for each of the test runs before the run you want to run.
View the suite run results
When the suite's tests run, the run results are compiled into a single report.
To view the suite run results:
-
In the test suite, select the Suite Runs tab.
-
In the list of suite runs, click the link for the suite run you want to view.
-
In the suite run instance, select the Runs tab. The list of all runs opens.
-
In the suite run, select the Report tab. The suite run report opens.
Tip: To view the test run details, click the test run ID.
-
In the View by box, select how to view the results:
- Tests: to show the results per tests
- Order: to show the results in the order of the runs
Scroll or click the links to view each test in the suite, as well as the specific steps in each test.
The report page also shows the phase each test was in when the run was created.
View additional details on suite runs
For automated tests, the Suite Run > Runs tab provides additional information.
Tab | Description |
---|---|
Preview |
Shows general information on the test run: Name, description, release assignment, and comments |
Report |
Provides a test run report after the run is executed. If there is an error, you will see the related stack trace information. |
Test Runners |
Shows details of the related test runners: Framework, CI server, job name, and build number. Click a test runner to filter its related test runs. When you run a suite, the test runner's status changes to Initializing, In Progress, and Finished. Click the toolbar refresh button to see the status changes. The tab also provides a link to access the test runner execution log. For details on configuring test runners, see Trigger on-demand automated test runs. |
See also: