Correlate dynamic values

This topic describes the process used to correlate dynamic data in your script.

In this topic:

About correlation

Web pages sent from the server to the browser often contain dynamic data that the client must send within later requests—for example, the session ID, which may contain a timestamp. This data changes every time you access the page.

After recording a business process, the script might contain dynamic values in the server responses. When the script is replayed, these recorded values are sent to the server. However, the web server rejects them because they are not valid in the replay session. Generally, the test fails when this happens.

To enable successful replay, the script must:

• Locate the dynamic data in the server responses, including headers.
• Extract and save that dynamic data.
• Replace the hard-coded dynamic data from the recording session with the data from the current replay session.

This process of locating, extracting, and replacing recorded dynamic values with values valid at replay is called correlation.

The DevWeb engine handles correlation during script generation by applying pre-defined correlation rules, or by performing a record scan on the recorded data.

Back to top

Correlation methods

Two methods are available for automatic correlation for dynamic values: rule-based scans and record-based scans. Both add correlations to the generated code using Extractors and appliers.

By default, the rules and record scans are disabled. You enable them in the generator_config.yml file (or in VuGen, in Recording Options > Code Generation).

Rule-based correlation

The DevWeb engine implements a set of predefined, extendable rules for correlating dynamic values in standard environments. These rules are defined in <correlation_rules>.yml files, in the correlation rules directory.

During code generation, the Offline Script Generator loads all activated rules found in the *.yml correlation rule files, and scans server responses for values that match the correlation rules. When a matching value is found, a step is added to the web request to extract the value and save it in a parameter.

Then, the script code is scanned for the value and matches are replaced with a reference to the parameter.

Use rule-based correlation:

In the correlation section of the <DevWeb root folder>\generator_config.yml file:

• Enable the rules scan by setting rulesScanEnabled to true.
• Change defaults for other correlation parameters as relevant, for example, the scan timeout or maximum length for the extracted value displayed in the script.

For details on defining correlation rule files, see Using correlation rules

Record-based correlation

The DevWeb engine provides automatic record-based correlation. When record scan is enabled, then during the code generation stage, the engine automatically scans the server responses for correlations. The search is based on internal engine configuration settings.

When a matching value is found, a step is added to the web request to extract the value and save it in a parameter, in the same way as for rules scan correlation. If the rules and the record scans are both enabled, the rules scan takes priority; this means that if both scans identify the same value, the engine applies the rule-based correlation.

The extractor type added by the record scan may be boundary or regular expression (according to the defined preferred extractor type), or an extractor appropriate for the detected content type, for example, an HTML, JSON, or XPath extractor.

Use record-based correlation:

1. In the correlation section of the <DevWeb root folder>\generator_config.yml file:

   • Enable the record scan by setting recordScanEnabled to true.
   • If required, change the preferred extractor type for plain text in the extractorType property: boundary or regexp.
   • If required, update the path in the recordScanConfigDirectory property.
   • Change defaults for other correlation parameters as relevant, for example, the scan timeout or maximum length for the extracted value displayed in the script.

2. Record the script. Extractors added by the record scan are indicated in the comment:

   

3. After code generation, you can use the generation log ..\<Script folder>\codegen to help identify if the candidates found are real correlations.

Back to top

Extractors and appliers

DevWeb engine correlations are based on extractors and appliers. The extractors can be added automatically during code generation based on rule-based or record-based scans; or added manually to the script.

When scanning server responses, the Offline Script Generator adds an extractor for each identified dynamic value. This enables the value to be extracted and saved in a parameter. Then, the script code is scanned for the value, and matches are replaced with a reference to the parameter.

• The extractors locate and extract the dynamic data in the responses. They can be based on boundary definitions or regular expressions, or can be used to query for object values inside JSON, XML, and HTML-formatted data. They also include converters for Base64, URL, and HTML encoding and decoding.

  Extractors can also be added to your scripts to validate and manipulate content. For example, you can add a text check extractor using the failOn value option.

  The available extractors include the following:

  • BoundaryExtractor(name, options) : ExtractorObject
  • RegexpExtractor(name, options) : ExtractorObject
  • JsonPathExtractor(name, options) : ExtractorObject
  • XpathExtractor(name, options) : ExtractorObject
  • TextCheckExtractor(name, options) : ExtractorObject
  • HtmlExtractor(name, options) : ExtractorObject
  • CookieExtractor(name, options) : ExtractorObject

• The appliers apply the extracted values, so that the saved parameter is used instead of the recorded value in subsequent requests to the server. You use the appliers to refine the way the extracted value is applied in the request. You can configure one or more appliers for each rule. See Extractors and appliers, below.

  If no appliers are defined for a rule, a default applier is used.

Applier types

The following table lists the applier types available.

The definitions for the supported parameters are provided here: Applier parameters

Applier parameters

The following gives the definitions for the applier parameters. To see which parameters are supported for each applier type, see Applier types, above.

Back to top

Using correlation rules

Rules-based correlation scans use the rules defined in correlation rule definition files, <correlation_rules>.yml. You create customized correlation rule files, to add and edit rules for your specific DevWeb scripts.

Correlation rule files are, by default, located in the <DevWeb root folder>\CorrelationRules\ folder. This folder includes a predefined correlation rules file for an ASP.NET testing environment, correlation_ASP_NET.yml. You can edit this file, or make copies to use as templates. Edit and extend the rule definitions as needed for your testing environment. You can also refer to the Example file below.

Each correlation rule group includes Extractors and appliers. Within the file you can enable or disable each rule group, and enable or disable the whole correlation rules file. If rules scan is enabled in generator_config.yml, then during code generation the Offline Script Generator scans all *.yml files in the CorrelationRules folder, and loads all enabled rules.

To use correlation rules:

1. Define correlation rule files (*.yml) for your testing environment. Save the files in the default <DevWeb root folder>\CorrelationRules\ folder, or in a different location.

2. Before recording your script, make sure that you have enabled all required correlation files and rule groups in the correlation *.yml files.
3. Configure correlation parameters in the <DevWeb root folder>\generator_config.yml file, including setting rulesScanEnabled to true, and modifying the rulesDirectory path if you are using a different folder for the correlation rule files. (See Rule-based correlation, above.)
4. Record the script. When a match is found for a rule, an extractor for the value is added to the script.

Example file

The following gives a example of an annotated <correlation_rules>.yml file:

categories:
  - name: category1                   #The name of the category, not used in code, this is for the user
    enabled: true                     #Allows the user to enable or disable all the rules in a category
    rules:                            #A list of rules
      - name: myRule1                 #The name of the rule, used in the code the user writes to identify the rule. Must be unique across all categories
        type: boundary                #The type of rule. Can be "boundary", "regexp", "json", "xpath"
        enabled: true                 #Allows the user to enable and disable particular rules
        options:                      #The options particular to this rule type
           leftBoundary: abc          #Left boundary of the boundary-based correlation
           rightBoundary: efg         #Right boundary of the boundary-based correlation
           caseInsensitive: false     #Search boundaries using case-insensitive comparison (default: true)
           scope: all                 #Scope define the required HTTP part: headers, body, or all Default: all
        appliers:
          - type: simple              #The type of the applier. One of "simple", "boundary", "regexp" or "none" which prevent apply automatically
            scope: headers            #Scope define the required HTTP part: url, querystring, headers, body, or all Default: all
            options:
              occurrence: first       #Occurrence focus on how many items to apply on all, first, last or valid number
              fieldName: ["cookie"]   #Field Name allow the applier to focus on one or more key entries
      - name: myRule2
        type: regexp
        enabled: true
        options:
           expression: "[0..9]*"      #The regular expression. See https://github.com/google/re2/wiki/Syntax for details
           flags: "im"                #The flags of the regular expression
           groupNumber: 0             #The group number (Default: 1 - for the first group)
           scope: body                #Scope define the required HTTP part: headers, body, or all Default: all
        appliers:
          - type: boundary
            scope: url
            options:
              leftBoundary: le>            #Left boundary of the boundary based correlation
              rightBoundary: </t            #Right boundary of the boundary based correlation
              caseInsensitive: false         #Search boundaries using case-insensitive comparison (default: true)
  - name: category2
    enabled: true
    rules:
      - name: myRule3
        type: json
        enabled: true
        options:
           path: abc                        #The JSONPath. See http://goessner.net/articles/JsonPath/ for details
        appliers:
          - type: regexp
            scope: all
            options:
              expression: "\\w+(.*?)\\w+"      #The regular expression see https://github.com/google/re2/wiki/Syntax for details
              flags: "i"                       #The flags of the regular expression, please note the group number is default 1 no need to define it
              occurrence: first
      - name: anotherRule3                     #Appliers section is optional on this case it will use the Default Applier
        type: xpath
        enabled: true
        options:
          path: //myEntry                      #The XPath, see https://godoc.org/gopkg.in/xmlpath.v2 for details

If no appliers are defined for a rule, the following default applier are used:

appliers:
  - type: simple
    scope: all
    options:
      occurrence: all

Back to top

See also:

• Script structure
• Offline Script Generator tool