Plan and run test suites

After you create a test suite, you can plan how to run it. This topic describes how to plan and run a test suite.

In this topic:

Plan a test suite run

Prepare the test suite to run. Assign owners and environment tags to each of the tests.

The test suite run process follows these steps:

To plan a test suite run:

1. On the Tests tab, open a test suite.

2. On the test suite's Planning tab, manage the tests you want to run.

   Action	Steps
   Add new or existing tests	Use the toolbar buttons to add new or existing tests. Tip: You can add the same test multiple times if, for example, you want to run the same test with different environment tags.
   Remove unwanted tests	Select the test's checkbox and select More > Remove from test suite.
   Select tests to include in the next run	Indicate which tests you want to include in the next run by turning on or off the switch in the Include in next run column. By default, all tests are set to run. Note: The checkboxes next to the test ID do not indicate whether to run the test. Selecting these checkboxes applies to other features, such as exporting, deleting, and reporting.
   Set the test run order	Drag and drop the tests to set the order in which they should be run. To send a test to the top or bottom of the list, right-click the test and select Rank Highest or Rank Lowest. The test run order indicates the priority of the tests in the suite, or the recommended order in which to run the tests. By default, the run order is not enforced. You can configure the suite to enforce the run order for manual tests. For details, see Enforce manual test run order.
   Add a test to a planned suite run	To add a test to a planned suite run, right-click the test and select Add to suite run. In the Add to Suite Run dialog box, select the suite runs to which you want to add the test, and click Add & Close. You can also add a run that already exists. This option is available for both manual and automated tests.

3. In the Planning tab, click the Plan Suite Run button . The Plan Suite Run dialog box lets you define the following suite's attributes.

   Field	Details
   Release	Select a release that you want to test. After you select a release, you can select a milestone or sprint, but they have different functions: Milestone: When you select a milestone, it means that the run contributes to the quality status of the milestone. A unique last run is created and assigned to the milestone for coverage and tracking purposes, similar to release. Sprint: When you select a sprint, it means that the run is planned to be performed in the time frame of the sprint. Sprints do not have their own unique run results, so filtering Last Runs by Sprint returns only runs that were not overridden in a later sprint. To see full run status and progress reports based on sprints, use the Test run history (manual) item type instead of using Last Runs in the Dashboard widgets. These values are assigned to all test runs for the tests in the suite. Tip: The values of user-defined fields are also assigned, if the user-defined fields have the same name and type in both the test suite run entity and its test run entities. This is available if the COPY_RUN_SUITE_UDFS_TO_RUNS configuration parameter is set to true.
   Program	(Optional) If you are working with programs, you can select a program to associate it with the suite. All of the runs in the suite are assigned to this program, which is recorded in each run’s history. For details, see Programs.
   Default environment tags and Default run by	Specify the Default environment tags and Default run by values to be used by all test suite runs. The test suite run's Environment tags and Run by fields inherit the default values. If the Default run by field is left empty, test runs are not assigned to any user. You need to manually assign each of the planned test runs to a user. Data access: If a test belongs to a data category that falls outside of the default tester's role, the run is not assigned to the default user. Manually assign the test run to an accredited user. For details, see Data access control. Note: If you modify either of these fields after planning a test suite, the changes only affect new runs. The existing runs continue to inherit the original default values.
   Enforce order for manual	When running a suite containing manual tests, set this field to Yes to enforce the run order of the tests in the suite. For details, see Enforce manual test run order. You can change the value of this field for a suite run only when creating the suite run. After a suite run has been created, this field becomes read-only and cannot be changed. Tip: This field might not be visible by default. To show it, click the Customize Fields button and select Enforce order for manual from the list.

4. In the Plan Suite Run box, click Plan.

   The suite run and test runs for the suite's tests are created or updated.

5. Users can run the planned tests from one of the following areas.

Area	Details
My Work module	Test runs are displayed in the user's item list in the My Work module. Each assigned run is labeled with the name of the suite run and a link to the individual test. Select the test and click the Run button to start the test run. If all tests in the suite are assigned to the same user, they appear in My Work as a suite run. Select the suite run and click the Run button .
Team Backlog or Quality modules	On the Details tab of the suite run, click the Run/Run Suite button . If you are running the entire test suite, and some tests are not assigned, indicate whether to run your assigned tests with the unassigned tests or all the tests in the test suite.

6. (Optional) You can export the test suite information in the Excel or CSV format.

   The export saves all fields, with the exception of UDFs (user-defined fields), and the following built-in fields: Author, Backlog coverage, BDD Spec, Builds, Component, Covered automated test, Covered manual test, Covered requirement, Data table, External test ID, Flag rules, Flags, Followed by me, In my work, Is draft, Is in latest version, Manual, Owner, Quality stories, and Test level.

Back to top

Run a test suite

A test suite runs as a collection of test runs.

By default, all the tests included in the test suite are run unless certain tests were excluded in the planning stage.

To run a test suite:

1. On the test suite Planning tab, verify that the necessary tests are included in the run.

   To change the tests included in the run, in the Include in next run column, select the tests to run. Note that checkboxes next to the test ID do not mark the tests to run.

2. Click the Run Suite button . The Run <test suite name> dialog box opens.

3. Verify the settings and click Let's Run.

   The Manual Runner window opens. For details, see Run manual and Gherkin tests.

The status of the test suite is automatically updated according to the status of the test runs.

Back to top

Manage suite runs

The suite run's Runs tab lets you manage the runs by changing their rank or other attributes. Use the right-click menu to add items to My Work or to send them as an email.

To manage the suite runs:

1. Select the Suite Runs tab. A list of all suite runs is displayed.

2. Click a suite run. The Runs tab opens and displays a list of runs.

3. To change the status of a run, modify the Native status in the Details tab and click Save. For example, if a run was previously blocked, you can change its status to Planned.

4. If you want to run a test twice as part of a suite run, make a copy of the run by selecting it and clicking Duplicate Run. If you want to redo a run from scratch, duplicate the run, make sure the copy's status is set to Planned (this is the default), and delete the old one.

5. Rank and reorder the runs by dragging them to the desired location.

   You cannot change the order of a run which is currently running.

   Tip: To ensure that the results stay aligned with their corresponding runs, keep the runs in the same order as they appear in the suite run, and do not run tests in parallel.

6. If you need to change the test assignment or an environment tag, modify the values in the Run by and Environment tag columns. The assignees receive a notification in their My Work area.

   Note: The Environment tag is an information label. The environment on which the test runs is not selected or detected.

7. After you assign all the tests, the Runs tab displays the updated list.

Back to top

Enforce manual test run order

When planning a run of a test suite containing manual tests, you can configure the suite to enforce the run order of the manual tests in the suite. This means that a manual test run cannot be run until the preceding manual test runs in the suite have been completed.

To enforce test run order for a suite run, do one of the following:

• Before creating the suite run, select the Details tab of the test suite and set the Enforce order for manual field to Yes.

  The value you set at the test suite level is the default for all future suite runs planned for the suite.

• When creating the suite run, set the Enforce order for manual field for the suite run to Yes. For details, see Plan a test suite run.

  Note: After a suite run has been created, the Enforce order for manual field becomes read-only and cannot be changed.

Test run completion status

By default, a test run is considered completed when its status is Passed, Failed, or Skipped. The administrator can choose which statuses are considered completed, by modifying the relevant business rule for the Manual Run entity. For details, see Business rules.

If a test run in the suite was completed, and its run status was then changed back to Planned after subsequent tests were already allowed to run, the updated status affects all subsequent test runs. This means that all test runs after the run marked Planned cannot be run.

To override this restriction and allow subsequent tests to run, you can manually change the Is completed field to Yes for each of the test runs before the run you want to run.

Back to top

View the suite run results

When the suite's tests run, the run results are compiled into a single report.

To view the suite run results:

1. In the test suite, select the Suite Runs tab.

2. In the list of suite runs, click the link for the suite run you want to view.

3. In the suite run instance, select the Runs tab. The list of all runs opens.

4. In the suite run, select the Report tab. The suite run report opens.

   Tip: To view the test run details, click the test run ID.

5. In the View by box, select how to view the results:

   • Tests: to show the results per tests
   • Order: to show the results in the order of the runs

   Scroll or click the links to view each test in the suite, as well as the specific steps in each test.

   The report page also shows the phase each test was in when the run was created.

Back to top

View additional details on suite runs

For automated tests, the Suite Run > Runs tab provides additional information.

Tab	Description
Preview	Shows general information on the test run: Name, description, release assignment, and comments
Report	Provides a test run report after the run is executed. If there is an error, you will see the related stack trace information.
Test Runners	Shows details of the related test runners: Framework, CI server, job name, and build number. Click a test runner to filter its related test runs. When you run a suite, the test runner's status changes to Initializing, In Progress, and Finished. Click the toolbar refresh button to see the status changes. The tab also provides a link to access the test runner execution log. For details on configuring test runners, see Trigger on-demand automated test runs.

Back to top

See also:

• Run manual and Gherkin tests
• Tags
• Create test suites