Plan and run test suites

After you create a test suite, you can plan how to run it. This topic describes how to plan and run a test suite.

Plan a test suite run

Prepare the test suite to run. Assign owners and environment tags to each of the tests.

The test suite run process follows these steps:

Tip: To apply the same plan to multiple test suites, in the tests grid, select the necessary test suites and click Plan Run .

To plan a test suite run:

  1. On the Tests tab, open a test suite.

  2. On the test suite's Planning tab, manage the tests you want to run:

    Action Steps

    Add new or existing tests

    Use the toolbar buttons to add new or existing tests.

    Tip: You can add the same test multiple times if, for example, you want to run the same test with different environment tags.
    Remove unwanted tests

    Select the test's check box and select More > Delete.
    Select tests to include in the next run

    Indicate which tests you want to include in the next run using the toggle in the Include in next run column. By default, all tests are set to run.

    Note: The check boxes next to the test ID do not indicate whether to run the test. Selecting these check boxes applies to other features, such as exporting, deleting, and reporting.
    Set the test run order

    Drag the tests to set the order in which they run.

    To send a test to the top or bottom of the list, right-click the test and select Rank Highest or Rank Lowest.
    Add a test to a planned suite run

    To add a test to a planned suite run, right-click the test and select Add to Suite Run. In the Add to Suite Run dialog box, select the suite runs to which you want to add the test, and click Add & Close.

    You can also add a run which already exists. This option is available for both manual and automated tests.

  3. In the Planning tab, click Plan Suite Run . The Plan Suite Run dialog box lets you define the following suite's attributes:

    Field Details
    Release

    Select a release that you want to test. After you select a release, you can select a milestone or sprint, but they have different functions:

    • Milestone: When you select a milestone, it means that the run contributes to the quality status of the milestone. A unique last run is created and assigned to the milestone for coverage and tracking purposes, similar to release.
    • Sprint: When you select a sprint, it means that the run is planned to be executed in the time frame of the sprint. Sprints do not have their own unique run results, so filtering Last Runs by Sprint returns only runs that were not overridden in a later sprint. To see full run status and progress reports based on sprints, use the Test run history (manual) item type instead of using Last Runs in the Dashboard widgets.

    ALM Octane assigns these values to all test runs for the tests in the suite.

    Tip: ALM Octane also assigns the values of user-defined fields, if the user-defined fields have the same name and type in both the test suite run entity and its test run entities. This is available if the COPY_RUN_SUITE_UDFS_TO_RUNS configuration parameter is set to true.
    Program (Optional) If you are working with programs, you can select a program to associate it with the suite. All of the runs in the suite are assigned to this program, which is recorded in each run’s history. For details, see Programs.
    Default environment tags and Default run by

    Specify the Default environment tags and Default run by to be used by all test suite runs.

    The test suite run's Environment tags and Run by fields inherit the default values. If the Default run by field is left empty, test runs are not assigned to any user. You need to manually assign each of the planned test runs to a user.

    Data access: If a test belongs to a data category that falls outside of the default tester's role, the run is not assigned to the default user. Manually assign the test run to an accredited user. For details, see Data access control.

    Note: If you modify the Default environment tags and Default Run by fields after planning a test suite, the changes only affect new runs. The existing runs continue to inherit the original default values.

  4. In the Plan Suite Run box, click Plan.

    ALM Octane creates or updates the suite run and test runs for the suite's tests.

  5. Users can run the planned tests from one of the following areas:

    6. Area Details
    My Work module

    ALM Octane displays test runs in the user's item list in the My Work module.

    ALM Octane labels each assigned run with the name of the suite run and a link to the individual test.

    Select the test, and in the right pane of the My Work module, click Run to start the test run.

    If all tests in the suite are assigned to the same user, they appear in My Work as a suite run. Select the suite run and click Run .
    Team Backlog or Quality modules

    On the Details tab of the suite run, click Run .

    If you are running the entire test suite, and some tests are not assigned, indicate whether to run your assigned tests with the unassigned tests or all the tests in the test suite.

Note: When you plan a suite run, it may take some time for you to see the planned runs, since they are created in an asynchronous manner.

Optionally, you can export the test suite information in the Excel or CSV format. The export saves all fields, with the exception of UDFs (user-defined fields), and the following built-in fields: Author, Backlog coverage, BDD Spec, Builds, Component, Covered automated test, Covered manual test, Covered requirement, Data table, External test ID, Flag rules, Flags, Followed by me, In my work, Is draft, Is in latest version, Manual, Owner, Quality stories, and Test level.

Back to top

Run a test suite

A test suite runs as a collection of test runs.

By default, all the tests included in the test suite are run unless certain tests were excluded in the planning stage.

Note: When a test suite runs, the included test runs are created in draft mode. The "Alert user" business rules that govern test runs creation are ignored.

You can prevent creating test runs in draft mode by setting the IGNORE_ALERT_RULE_WHEN_CREATE_IN_DRAFT_MODE parameter to false. As a result, the "Alert user" business rules are triggered, and the runs cannot be created as drafts. When a test suite runs, an error occurs.

For details, see Configuration parameters.

To run a test suite:

  1. On the test suite Planning tab, verify that the necessary tests are included in the run.

    To change the tests included in the run, in the Include in next run column, toggle the tests to run. Note that check boxes next to the test ID do not mark the tests to run.

  2. Click Run Suite . The Run <test suite name> dialog box opens.

  3. Verify the settings and click Let's Run.

    The Manual Runner window opens. For details, see Plan and run manual and Gherkin tests.

The status of the test suite is automatically updated according to the status of the tests runs.

Note: The final status of the test suite follows ALM Octane's business rule logic. The business rules run sequentially and the final status is dictated by the last rule. For example, the first built-in business rule sets the test suite status to Passed, if a test suite's tests were either Skipped or Passed. However, a subsequent rule whose criteria is to look for any Skipped tests, resets the status of the test suite to Skipped. The business rules and their order are set by the admin. For details, see Business rules.

Back to top

Manage suite runs

The suite run's Runs tab lets you manage the runs by changing their rank or other attributes. Use the right-click menu to add items to My Work or to send them as an email.

Tip: To update details for multiple test runs, do a multiple selection and choose Bulk Update from the context menu.

To manage the suite runs:

  1. Select the Suite Runs tab. ALM Octane displays a list of all suite runs.

  2. Click on a suite run. ALM Octane opens the Runs tab and displays a list of runs.

  3. To change the status of a run, modify the Native status in the Details tab and click Save. For example, if a run was previously blocked, you can change its status to Planned.

  4. If you want to run a test twice as part of a suite run, make a copy of the run by selecting it and clicking Duplicate Run. If you want to redo a run from scratch, duplicate the run, make sure the copy's status is set to Planned (this is the default), and delete the old one.

  5. Rank and reorder the runs by dragging them to the desired location.

    You cannot change the order of a run which is currently running.

    Tip: To ensure that the results stay aligned with their corresponding runs, keep the runs in the same order as they appear in the suite run, and do not run tests in parallel.

  6. If you need to change the test assignment or an environment tag, modify the values in the Run by and Environment tag columns. The assignees receive a notification in their My Work area.

    Note: Environment tag is an information label. ALM Octane does not select or detect the environment on which the test runs.

  7. After you assign all the tests, the Runs tab displays the updated list. The Smart List view is shown below:

Back to top

View the suite run results

When the suite's tests run, ALM Octane compiles the run results into a single report.

To view the suite run results:

  1. In the test suite, click the Suite Runs tab.

  2. In the list of suite runs, click the link for the suite run you want to view.

  3. In the suite run instance, click the Runs tab. ALM Octane displays the list of all runs.

  4. In the suite run, click the Report tab. The suite run report is displayed.

    Tip: To view the test run details, click the test run ID.

  5. In the View by box (top right), select how to view the results:

    • Tests: to show the results per tests
    • Order: to show the results in the order of the runs

    Scroll or click the links to view each test in the suite, as well as the specific steps in each test.

    The report page also shows the phase each test was in when the run was created.

Back to top

View additional details on suite runs

For automated tests, the right pane of the Suite Run > Runs tab provides additional information:

  • The Preview tab shows general information on the test run: Name, description, release assignment, and comments.
  • The Report tab provides a test run report after the run is executed. If there is an error, you will see the related stack trace information.

  • The Test Runners tab shows details of the related test runners: Framework, CI server, job name, and build number. Click a test runner to filter its related test runs.

    When you run a suite, the test runner's status changes to Initializing, In Progress, and Finished. Click the toolbar refresh button to see the status changes. The tab also provides a link to access the test runner execution log.

    For details on configuring test runners, see Trigger on-demand automated test runs.

Back to top

See also: