Chaos testing

You can add disruption events from chaos engineering platforms to your performance tests. These events provide simulated attacks on your services and environments.

In this topic:

About chaos testing

Chaos testing enables you to see how your system responds under stress. By incorporating simulated attacks from chaos engineering platforms into your performance tests, you can determine the impact of potential failures on your applications.

The simulated attacks can target different technologies and components such as databases, web servers, CPU, and memory. To trigger the attacks, you add disruption events to your test. When you run the test, the microservice component, EventsHandlerMicroservice, sends a request to the chaos platform to run the predefined attacks. This enables the attacks to launch on the system in parallel with the test.

An attack usually impacts the typical workflow, limiting responses or reducing performance. For example, the web server may work slower than usual, and there may be fewer successful transactions. You can compare responses when your test runs without attacks and when the system is under stress.

Chaos engineering platforms

OpenText Enterprise Performance Engineering works with Steadybit on public cloud and on-premises private cloud platforms. In Steadybit, you define experiments with multiple attacks and add those experiments to your test.

The Steadybit agent must run on the AUT machines. Integrated Steadybit APIs are then used to coordinate the chaos testing.

When you work with Steadybit, define multiple attacks in one experiment instead of running multiple shorter experiments one after the other.

Add disruption event provider

You connect the attack scenarios from your disruption events provider to performance tests, to run in parallel with the scripts.

To integrate Steadybit tests:

1. Set up and configure your Steadybit environment for the application you are testing. Make sure to use Steadybit team access tokens which map to the permissions of a team member.

2. Test the connection between the agent and the chaos platform before running the performance test.

3. In the Administration module, select Configuration > Integrations > Disruption Providers.

4. Open the Steadybit pane and click the Add Disruption Provider button .

5. In the New Provider page, enter a logical name for the disruption events provider.

6. Enter the API Key of the provider. This key is saved encrypted.

7. Select the provider's location:

   • If working with a private cloud platform, select On-Prem and enter the server URL (IP address and port if relevant).

   • If working with the default environment, select Default. The default server URL is used: https://platform.steadybit.com/api. This field is read-only. The default URL is not hard-coded and must be configured in the internal configuration.

8. (Optional) Enter the team and description of the event provider.

9. To test the connection to the Steadybit environment, click Test Connection.

   Note: Test connection is only available when all required data has been filled.

10. Click Save to add the provider.

Assign disruption event provider to projects

You can assign disruption providers to different projects.

To assign an event provider to projects:

1. Prerequisite: Integrate an event provider. For details, see Add disruption event provider.

2. In the Linked Projects area, click the Assign button to open the Assign Projects dialog box.

3. Select the projects to which you want to assign the disruption events provider, and click Assign. The selected projects are added to the Linked Projects grid.

4. You can view project details such as the project ID, name, and domain, and add or remove linked projects using the Linked Projects grid.

Add disruption events to a test

You can add disruption events to a performance test from the Disruption Events tab in the performance test designer.

To add disruption events to a test:

1. Prerequisites:

   • Integrate an event provider. For details, see Add disruption event provider.

   • Assign an event provider to the project. For details, see Assign disruption event provider to projects.

2. Click the Navigation menu and select Test Management > Tests & Scripts.

3. Select a performance test in the test management tree, and click Edit Test.

4. In the Disruption Events tab, click the Add Events button . The Add Events dialog box opens and lists available events from the chaos server assigned to the project, including the event name, attack type, and event creator.

5. Select the disruption events you want to run for the current test from the grid.

   • You can use the Search button to search for events by name. This filters the list by your search term. Events that have already been added to the test are not listed (to prevent adding duplicate events).

   • You can only select events that have Active state.

   • You can hover over an experiment name to see a description, if a description was added in Steadybit. You can also hover over an experiment type, to see a list of all actions it contains.

   Click Add to add the selected events to the events grid. The grid displays the event name, platform, start time, and duration.

6. (Optional) Define the required start time for each event.

   Tip: We recommend setting a different start time for each event, and to avoid scheduling events to overlap because this can result in unexpected behavior. Duration can only be changed from the Steadybit platform.

   1. To schedule the start time for an event, double-click the event's Start Time and edit it as required.

      To set the required start time for multiple events, select the events and click the Update Start Time button in the toolbar. Enter a start time in the Start time box, and click Apply. The start time is applied to all the selected events.

      The start time is relative to the overall test scenario start time. The end time automatically adjusts when you modify the start time, based on the duration of the event.

   2. The scheduling graph shows when each selected event is due to start and end, against the global Vusers schedule for the performance test. Select an event from the legend to highlight the event lines in the graph.

      

   Note:  

   • The smallest time unit that is supported for disruption events is seconds.

   • All events that are present are added to the test unless they are inactive, already running, or missing. Events that are already running or are missing are emphasized in the graph.

7. Click Save.

   Note: A warning is displayed if:

   • you select duplicate events in the list (events running the same attacks).

   • events overlap, which may result in one event disrupting the other.

   • the end time for an event is after the performance test run time. In this case, the event is forcefully stopped when the test run ends.

   The events will still run despite the warning, if you choose to leave your configuration as defined.

8. Run the performance test.

   During the test run, the Disruption Events graph displays bars for each event start and end time so you can see how the chaos event impacts the monitored machine.

   Click on the bars to highlight them and display the event name.

   Note: In the graph settings, you can configure to display or hide the event bars for each graph. You can also display or hide for all graphs in the global graph settings.

   

   The Messages view shows notifications when each disruption event starts and stops and reports errors for the event run. For details, see Messages view.

9. After the performance test run, view the graph in the offline results or in Analysis. The Chaos Events graph is automatically displayed, showing data for the disruption events.

   The event start and end times are also shown in the Summary Report. For details, see HTML Report.

Remove disruption events from a test

You can remove disruption events from a performance test.

To remove events from a test:

1. Click the Navigation menu and select Test Management > Tests & Scripts.

2. Select the performance test in the test management tree from which you want to remove events, and click Edit Test.

3. Open the Disruption Events tab.

4. Select the events that you want to remove from the test and click the Delete button .

Deactivate a disruption event

You can deactivate an event to make it temporarily unavailable in a test.

For example, you might want to run a specific test with simulated chaos attacks, and then deactivate the event to run the same test without those attacks.

To deactivate a disruption event:

1. Click the Navigation menu and select Test Management > Tests & Scripts.

2. In the test management tree, select the performance test from which you want to deactivate events, and click Edit Test.

3. Open the Disruption Events tab.

4. Select the events that you want to deactivate from the test and click the Deactivate button .

5. Click OK to confirm. The selected events are deactivated and appear unavailable in the Events list, but are not deleted from the server.

   To re-activate events, repeat the previous step and click the Activate Event button .

Disruption event logs

You can view the logs for the disruption events:

• Microservice-related logs: <LRE_install_dir>\Logs\EventsHandlerSrv\events_handler.log

• Controller-related logs: %TEMP%\lr_disruption_events_err.log

Logging levels are set in these files:

• %LR_PATH%\bin\disruption\*log

• %LR_PATH%\bin\disruption\*config

The following log levels are listed in increasing priority.

.*config files	.*json files	Recommended values
All	Trace	Not Applicable
Debug	Debug	For debugging
Info	Information	Not Applicable
Warn	Warning	For regular workflow
Error	Error	Not Applicable
Fatal	Critical	Not Applicable
Off	None	Not Applicable

For more information, see the official documentation for Apache log4net and for .NET Core.

Notes and limitations

This section describes notes and limitations when using chaos testing in performance tests.

• Tests configured to run with a Goal-Oriented schedule do not support chaos tests; the Disruption Providers tab is unavailable.

• Events are not displayed in the graph until their duration ends. For example, an event that starts in the second minute and lasts three minutes is not displayed until the fifth minute of the scheduler.

See also

• Tests
• Performance Test Designer