Share this page

# Correlate dynamic values

This topic describes the process used to correlate dynamic data in your script.

In this topic:

## About correlation

Web pages sent from the server to the browser often contain dynamic data that the client must send within later requests—for example, the session ID, which may contain a timestamp. This data changes every time you access the page.

After recording a business process, the script might contain dynamic values in the server responses. When the script is replayed, these recorded values are sent to the server. However, the web server rejects them because they are not valid in the replay session. Generally, the test fails when this happens.

To enable successful replay, the script must:

- Locate the dynamic data in the server responses, including headers.
- Extract and save that dynamic data.
- Replace the hard-coded dynamic data from the recording session with the data from the current replay session.

This process of locating, extracting, and replacing recorded dynamic values with values valid at replay is called correlation.

The DevWeb engine handles correlation during script generation by applying pre-defined correlation rules, or by performing a record scan on the recorded data.

Two methods are available for automatic correlation for dynamic values: rule-based scans and record-based scans. Both add correlations to the generated code using Extractors and appliers.

By default, the rules and record scans are disabled. You enable them in the generator_config.yml file (or in VuGen, in Recording Options > Code Generation).

Back to top

## Rule-based correlation

The DevWeb engine implements a set of predefined, extendable rules for correlating dynamic values in standard environments. These rules are defined in <correlation_rules>.yml files, in the correlation rules directory.

During code generation, the Offline Script Generator loads all activated rules found in the *.yml correlation rule files, and scans server responses for values that match the correlation rules. When a matching value is found, a step is added to the web request to extract the value and save it in a parameter.

Then, the script code is scanned for the value and matches are replaced with a reference to the parameter.

Use rule-based correlation:

In the correlation section of the <DevWeb root folder>\generator_config.yml file:

- Enable the rules scan by setting rulesScanEnabled to `true`.
- Change defaults for other correlation parameters as relevant, for example, the scan timeout or maximum length for the extracted value displayed in the script.

For details on defining correlation rule files, see Using correlation rules

Back to top

## Record-based correlation

The DevWeb engine provides automatic record-based correlation. When record scan is enabled, then during the code generation stage, the engine automatically scans the server responses for correlations. The search is based on internal engine configuration settings.

When a matching value is found, a step is added to the web request to extract the value and save it in a parameter, in the same way as for rules scan correlation. If the rules and the record scans are both enabled, the rules scan takes priority; this means that if both scans identify the same value, the engine applies the rule-based correlation.

The extractor type added by the record scan may be boundary or regular expression (according to the defined preferred extractor type), or an extractor appropriate for the detected content type, for example, an HTML, JSON, or XPath extractor.

Note: During the record scan, predefined settings are used to filter content and patterns, to help detect the correlation candidates. These configurations are defined in the <DevWeb root folder>\resources\CorrelationEngine\record_scan_config.yml file.

We recommend that you do not make changes to this file. If you do need to make changes, copy the record_scan_config.yml file to a new location and make the changes in the copy. Then change the path in the recordScanConfigDirectory key, as described below.

To use record-based correlation:

1. In the correlation section of the <DevWeb root folder>\generator_config.yml file:

   - Enable the record scan by setting recordScanEnabled to `true`.
   - If required, change the preferred extractor type for plain text in the extractorType property: boundary or regexp.
   - If required, update the path in the recordScanConfigDirectory property.
   - Change defaults for other correlation parameters as relevant, for example, the scan timeout or maximum length for the extracted value displayed in the script.

2. Record the script. Extractors added by the record scan are indicated in the comment:

3. After code generation, you can use the generation log ..\<Script folder>\codegen to help identify if the candidates found are real correlations.

Back to top

## Extractors and appliers

DevWeb engine correlations are based on extractors and appliers. The extractors can be added automatically during code generation based on rule-based or record-based scans; or added manually to the script.

When scanning server responses, the Offline Script Generator adds an extractor for each identified dynamic value. This enables the value to be extracted and saved in a parameter. Then, the script code is scanned for the value, and matches are replaced with a reference to the parameter.

Tip: When the log level (in rts.yml) is set to debug or trace, the extractor substitutions are reported in the log. The log messages document each place in the script that the extractor is applied, and the substitute value used for each instance.

- The extractors locate and extract the dynamic data in the responses. They can be based on boundary definitions or regular expressions, or can be used to query for object values inside JSON, XML, and HTML-formatted data. They also include converters for Base64, URL, and HTML encoding and decoding.

  Extractors can also be added to your scripts to validate and manipulate content. For example, you can add a text check extractor using the failOn value option.

  The available extractors include the following:

  - BoundaryExtractor(name, options) : ExtractorObject
  - RegexpExtractor(name, options) : ExtractorObject
  - JsonPathExtractor(name, options) : ExtractorObject
  - XpathExtractor(name, options) : ExtractorObject
  - TextCheckExtractor(name, options) : ExtractorObject
  - HtmlExtractor(name, options) : ExtractorObject
  - CookieExtractor(name, options) : ExtractorObject

- The appliers apply the extracted values, so that the saved parameter is used instead of the recorded value in subsequent requests to the server. You use the appliers to refine the way the extracted value is applied in the request. You can configure one or more appliers for each rule. See Extractors and appliers, below.

  If no appliers are defined for a rule, a default applier is used.

Applier types

The following table lists the applier types available.

The definitions for the supported parameters are provided here: Applier parameters

Applier type | Description | Supported parameters
--- | --- | ---
simple | Replaces the extracted value with a dynamic parameter that was created from the extractor. The replacement can occur in any subsequent requests to the original response (the server response containing the extractor). | occurrence (optional) scope (optional) fieldName (optional)
boundary | Replaces an extracted value between defined boundaries (left and right), with a dynamic parameter that was created from the extractor. The replacement can occur in any subsequent requests to the original response (the server response containing the extractor). | leftBoundary (mandatory) rightBoundary (mandatory) occurrence (optional) scope (optional) caseInsensitive (optional) fieldName (optional)
regexp | Replaces an extracted value that follows a regular expression, with a dynamic parameter that was created from the extractor. The replacement can occur in any subsequent requests to the original response (the server response containing the extractor). | expression (mandatory) flags (optional) occurrence (optional) scope (optional) groupNumber (optional) fieldName (optional)
none | Used when you want to enable the generation of extractors in the script, but do not want the extracted values to be applied in the script. This avoids the default behavior: If DevWeb cannot find where to apply the extracted values, then the extractor is not generated. | n/a

Applier parameters

The following gives the definitions for the applier parameters. To see which parameters are supported for each applier type, see Applier types, above.

Parameter | Data type | Description
--- | --- | ---
expression | string | The regular expression.
flags | string | The flags for the regular expression.
leftBoundary | string | The left boundary.
rightBoundary | string | The right boundary.
caseInsensitive | boolean | Search boundaries using non-case sensitive comparison. Default: true
occurrence | string/number | The occurrence of the value to replace: all. All the found occurrences are replaced. first. The first occurrence is replaced. last. The last occurrence is returned. A number defining the position of the value to replace, where 0 defines the first occurrence. So, for example, 2 replaces the third occurrence. If occurrence is not defined, the first occurrence is replaced.
scope | string | The relevant HTTP part in which the value will be replaced: url, querystring, headers, body, or all Default: all
fieldName | list of strings | A key (or keys) defining the focus within the scope area. For example, if the value for scope is `headers`, but you want the replacement to be in a particular header MyHeader, set fieldName to `[“MyHeader”]`.

Back to top

## Using correlation rules

Rules-based correlation scans use the rules defined in correlation rule definition files, <correlation_rules>.yml. You create customized correlation rule files, to add and edit rules for your specific DevWeb scripts.

Correlation rule files are, by default, located in the <DevWeb root folder>\CorrelationRules\ folder. This folder includes a predefined correlation rules file for an ASP.NET testing environment, correlation_ASP_NET.yml. You can edit this file, or make copies to use as templates. Edit and extend the rule definitions as needed for your testing environment. You can also refer to the Example yml file below.

Each correlation rule group includes Extractors and appliers. Within the file you can enable or deactivate each rule group, and enable or deactivate the whole correlation rules file. If rules scan is enabled in generator_config.yml, then during code generation the Offline Script Generator scans all *.yml files in the CorrelationRules folder, and loads all enabled rules.

Tip: Check out the video: Tutorial #7: Correlations.

To use correlation rules:

1. Define correlation rule files (*.yml) for your testing environment. Save the files in the default <DevWeb root folder>\CorrelationRules\ folder, or in a different location.

2. Before recording your script, make sure that you have enabled all required correlation files and rule groups in the correlation *.yml files.
3. Configure correlation parameters in the <DevWeb root folder>\generator_config.yml file, including setting rulesScanEnabled to `true`, and modifying the rulesDirectory path if you are using a different folder for the correlation rule files. (See Rule-based correlation, above.)
4. Record the script. When a match is found for a rule, an extractor for the value is added to the script.

Back to top

## Example yml file

The following example shows an annotated <correlation_rules>.yml file:

    categories:
      - name: category1                   #The name of the category, not used in code, this is for the user
        enabled: true                     #Allows the user to enable or disable all the rules in a category
        rules:                            #A list of rules
          - name: myRule1                 #The name of the rule, used in the code the user writes to identify the rule. Must be unique across all categories
            type: boundary                #The type of rule. Can be "boundary", "regexp", "json", "xpath"
            enabled: true                 #Allows the user to enable and disable particular rules
            options:                      #The options particular to this rule type
               leftBoundary: abc          #Left boundary of the boundary-based correlation
               rightBoundary: efg         #Right boundary of the boundary-based correlation
               caseInsensitive: false     #Search boundaries using case-insensitive comparison (default: true)
               scope: all                 #Scope define the required HTTP part: headers, body, or all Default: all
            appliers:
              - type: simple              #The type of the applier. One of "simple", "boundary", "regexp" or "none" which prevent apply automatically
                scope: headers            #Scope define the required HTTP part: url, querystring, headers, body, or all Default: all
                options:
                  occurrence: first       #Occurrence focus on how many items to apply on all, first, last or valid number
                  fieldName: ["cookie"]   #Field Name allow the applier to focus on one or more key entries
          - name: myRule2
            type: regexp
            enabled: true
            options:
               expression: "[0..9]*"      #The regular expression. See https://github.com/google/re2/wiki/Syntax for details
               flags: "im"                #The flags of the regular expression
               groupNumber: 0             #The group number (Default: 1 - for the first group)
               scope: body                #Scope define the required HTTP part: headers, body, or all Default: all
            appliers:
              - type: boundary
                scope: url
                options:
                  leftBoundary: le>            #Left boundary of the boundary based correlation
                  rightBoundary: </t            #Right boundary of the boundary based correlation
                  caseInsensitive: false         #Search boundaries using case-insensitive comparison (default: true)
      - name: category2
        enabled: true
        rules:
          - name: myRule3
            type: json
            enabled: true
            options:
               path: abc                        #The JSONPath. See http://goessner.net/articles/JsonPath/ for details
            appliers:
              - type: regexp
                scope: all
                options:
                  expression: "\\w+(.*?)\\w+"      #The regular expression see https://github.com/google/re2/wiki/Syntax for details
                  flags: "i"                       #The flags of the regular expression, please note the group number is default 1 no need to define it
                  occurrence: first
          - name: anotherRule3                     #Appliers section is optional on this case it will use the Default Applier
            type: xpath
            enabled: true
            options:
              path: //myEntry                      #The XPath, see https://godoc.org/gopkg.in/xmlpath.v2 for details

If no appliers are defined for a rule, the following default applier are used:

    appliers:
      - type: simple
        scope: all
        options:
          occurrence: all

Back to top

See also:

- Script structure
- Offline Script Generator tool

https://admhelp.microfocus.com Performance Engineering for Developers 8942
Tips to improve your search results... OpenText™ Performance Engineering for Developers Help Center OpenText™ Performance Engineering for Developers Help 26.1 All View Local Help Center View Online Help Center Version: