Plan and run test suites
After you create a test suite, you plan how to run it. This topic describes how to plan and run a test suite.
Prepare the test suite to be run. Assign owners and environments to each of the tests.
The test suite run process follows these steps:
To plan a test suite run:
- In the Tests tab, click on a test suite.
In the test suite's Planning tab, manage the tests you want to run:
Use the toolbar buttons to add new or existing tests.
Tip: You can add the same test multiple times, if, for example, you want to run the same test on different environments.
Remove any unwanted tests. Select the test's check box and choose More > Delete.
Indicate which tests you want to include in the next run using the toggler in the Include in next run column. By default, all tests are set to run.
Note: The check boxes adjacent to the test ID do not indicate whether to run the test. Selecting these check boxes applies to other features, such as exporting, deleting, and reporting.
- Drag the tests to set the order in which they will run. To send a test to the top or bottom of the list, right-click the test and choose Rank Highest or Rank Lowest.
- To add a test to a planned suite run, right-click the test and select Add to Suite Run. In the Add to Suite Run dialog box, select the suite runs to which you want to add the test, and click Add & Close. You can also add a run which already exists. (This is available for both manual and automated tests.)
In the Planning tab, Click Plan Suite Run. The Plan Suite Run dialog box lets you define the suite's attributes.
Tip: To apply the same plan to multiple test suites, select the test suites and click Plan Suite Run.
Select a release that you want to test. After you select a release, you can select a milestone or sprint, but they fill different functions:
Milestone. When you select a milestone, it means that the run contributes to the quality status of the milestone. A unique last run will be created and assigned to the milestone for coverage and tracking purposes, similar to release.
Sprint. When you select a sprint, it means that the run is planned to be executed in the time frame of the sprint. Sprints do not have their own unique run results, so filtering Last Runs by Sprint will only return runs that were not overridden in a later sprint. To see full run status and progress reports based on sprints, use the Test run history (manual) item type instead of using Last Runs in the Dashboard widgets.
ALM Octane assigns these values to all test runs for the tests in the suite.
Tip: ALM Octane also assigns the values of user-defined fields, if the user-defined fields have the same name and type in both the test suite run entity and its test run entities. This is available if the COPY_RUN_SUITE_UDFS_TO_RUNS configuration parameter is set to true.
(Optional) If you are working with programs, you can select a program to associate it with the suite. All of the runs in the suite will be assigned this program, and it is added to each run’s history. For details, see Programs (Enterprise Edition).
Specify the Default environment and Default run by to be used by all test suite runs. The test suite run's Environment and Run by fields inherit the default values. If the Default run by field is left empty, test runs are not assigned to any user. You will need to manually assign each of the planned test runs to a user.
Data access: If a test belongs to a data category that falls outside of the default tester's role, the run will not be assigned to the default user. Manually assign the test run to an accredited user. For details, see Data access (Enterprise Edition).
Note: If you modify the Default Environment and Default Run by fields after planning a test suite, existing runs will continue to inherit the original default values. The changes will only affect new runs.
Owners run the tests from one of the following:
My Work module
ALM Octane labels each assigned run with the name of the suite run and a link to the individual test:
Select the test, and in the right pane of the My Work module, click Run Suite to start the test run.
Tip: If all tests in the suite are assigned to the same user, they appear in My Work as a suite run. Select the suite and, in the right pane, click the Run Suite button.
From the Team Backlog or Quality modules
In the Details tab of the suite run, click the Run button .
If you are running the entire suite run, and some tests are not assigned, indicate whether to run your assigned tests with the unassigned tests or all tests in the test suite.
- Optionally, export the test suite information to Excel. The export saves all fields to a CSV file, with the exception of UDF (user-defined fields), and the following built-in fields: Author, Backlog coverage, BDD Spec, Builds, Component, Covered automated test, Covered manual test, Covered requirement, Data table, External test ID, Flag rules, Flags, Followed by me, In my work, Is draft, Is in latest version, Manual, Owner, Quality stories, and Test level.
- In the Plan Suite Run box, click Plan. ALM Octane creates the suite run and test runs for the suite's tests. ALM Octane displays the test runs in the owner's item list in the My Work module.
Note: When you plan a suite run, it may take some time for you to see the planned runs, since they are created in an asynchronous manner.
When you run a test suite, the test suite runs as a collection of test runs.
To run a test suite:
In the test suite, click Run Suite. The Run <test suite name> dialog opens.
Verify the settings and click LET'S RUN. The Manual Runner window opens. For details, see Run manual and Gherkin tests.
The status of the test suite is automatically updated according to the status of the tests runs.
Note: The final status of the test suite follows ALM Octane's business rule logic. The business rules run sequentially and the final status is dictated by the last rule. For example, the first built-in business rule sets the test suite status to Passed, if a test suite's tests were either Skipped or Passed. However, a subsequent rule whose criteria is to look for any Skipped tests, will reset the status of the test suite to Skipped. The business rules and their order are set by the admin. For details, see Business rules.
The suite run's Runs tab lets you manage the runs by changing their rank or other attributes. Use the right-click menu to add items to My Work or to send them as an email.
Tip: To update details for multiple test runs, do a multiple selection and choose Bulk Update from the context menu.
To manage the suite runs:
Select the Suite Runs tab. ALM Octane displays a list of all suite runs.
Click on a suite run. ALM Octane opens the Runs tab and displays a list of runs.
To change the status of a run, modify the Native status in the Details tab and click Save. For example, if a run was previously blocked, you can change its status to Planned.
If you want to run a test twice as part of a suite run, make a copy of the run by selecting it and clicking Duplicate Run. If you want to redo a run from scratch, duplicate the run, make sure the copy's status is set to Planned (this is the default), and delete the old one.
Rank and reorder the runs by dragging them to the desired location.
You cannot change the order of a run which is currently running.
Tip: Results of runs must be in the same order that the runs appear on the suite run, to prevent switches between runs results. We recommending therefore to not run tests in parallel.
If you need to change the test assignment or environment, modify the values in the Run by and Environment columns. The assignees receive a notification in their My Work area.
Note: Environment is an information label. ALM Octane does not select or detect the environment on which the test runs.
After you assign all the tests, the Runs tab displays the updated list. The Smart List view is shown below:
When the suite's tests run, ALM Octane compiles the run results into a single report.
To view the suite run results:
In the test suite, click the Suite Runs tab.
In the list of suite runs, click on the link for the suite run you want to view.
In the suite run instance, click on the Runs tab. ALM Octane displays the list of all runs.
In the suite run, click the Report tab. The suite run report is displayed:
In the View by box (top right), select how to view the results: Tests to show the results per tests, or Order, to show the results in the order of the runs.
Scroll or click the links to view each test in the suite, as well as the specific steps in each test.
The report page also shows the phase each test was in when the run was created.
View additional details on suite runs
For automated tests, the right pane of the Suite Run > Runs tab provides additional information:
- The Preview tab shows general information on the test run: Name, description, release assignment, and comments.
- The Report tab provides a test run report after the run is executed. If there is an error you will see the related stack trace information.
The Test Runners tab shows details of the related test runners: Framework, CI server, job name, and build number. Click a test runner to filter its related test runs.
When you run a suite, the test runner's status changes to Initializing, In Progress, and Finished. Click the toolbar refresh button to see the status changes. The tab also provides a link to access the test runner execution log.
For details on configuring test runners, see Trigger automated test runs in your testing framework.