Correlate dynamic values
This topic describes the process used to correlate dynamic data in your script.
About correlation
Web pages sent from the server to the browser often contain dynamic data that the client must send within later requests—for example, the session ID, which may contain a timestamp. This data changes every time you access the page.
After recording a business process, the script might contain dynamic values in the server responses. When the script is replayed, these recorded values are sent to the server. However, the web server rejects them because they are not valid in the replay session. Generally, the test fails when this happens.
To enable successful replay, the script must:
- Locate the dynamic data in the server responses, including headers.
- Extract and save that dynamic data.
- Replace the hard-coded dynamic data from the recording session with the data from the current replay session.
This process of locating, extracting, and replacing recorded dynamic values with values valid at replay is called correlation.
The DevWeb engine handles correlation during script generation by applying pre-defined correlation rules, or by performing a record scan on the recorded data.
Correlation methods
Two methods are available for automatic correlation for dynamic values: rule-based scans and record-based scans. Both add correlations to the generated code using Extractors and appliers.
By default, the rules and record scans are disabled. You enable them in the generator_config.yml file (or in VuGen, in Recording Options > Code Generation).
The DevWeb engine implements a set of predefined, extendable rules for correlating dynamic values in standard environments. These rules are defined in <correlation_rules>.yml files, in the correlation rules directory.
During code generation, the Offline Script Generator loads all activated rules found in the *.yml correlation rule files, and scans server responses for values that match the correlation rules. When a matching value is found, a step is added to the web request to extract the value and save it in a parameter.
Then, the script code is scanned for the value and matches are replaced with a reference to the parameter.
Use rule-based correlation:
In the correlation section of the <DevWeb root folder>\generator_config.yml file:
- Enable the rules scan by setting rulesScanEnabled to
true
. - Change defaults for other correlation parameters as relevant, for example, the scan timeout or maximum length for the extracted value displayed in the script.
For details on defining correlation rule files, see Using correlation rules
Record-based correlation
The DevWeb engine provides automatic record-based correlation. When record scan is enabled, then during the code generation stage, the engine automatically scans the server responses for correlations. The search is based on internal engine configuration settings.
When a matching value is found, a step is added to the web request to extract the value and save it in a parameter, in the same way as for rules scan correlation. If the rules and the record scans are both enabled, the rules scan takes priority; this means that if both scans identify the same value, the engine applies the rule-based correlation.
The extractor type added by the record scan may be boundary or regular expression (according to the defined preferred extractor type), or an extractor appropriate for the detected content type, for example, an HTML, JSON, or XPath extractor.
Note: During the record scan, LoadRunner Developer uses predefined settings to filter content and patterns, to help detect the correlation candidates. These configurations are defined in the <DevWeb root folder>\resources\CorrelationEngine\record_scan_config.yml file.
We recommend that you do not make changes to this file. If you do need to make changes, copy the record_scan_config.yml file to a new location and make the changes in the copy. Then change the path in the recordScanConfigDirectory key, as described below.
Use record-based correlation:
-
In the correlation section of the <DevWeb root folder>\generator_config.yml file:
- Enable the record scan by setting recordScanEnabled to
true
. - If required, change the preferred extractor type for plain text in the extractorType property: boundary or regexp.
- If required, update the path in the recordScanConfigDirectory property.
- Change defaults for other correlation parameters as relevant, for example, the scan timeout or maximum length for the extracted value displayed in the script.
- Enable the record scan by setting recordScanEnabled to
-
Record the script. Extractors added by the record scan are indicated in the comment:
- After code generation, you can use the generation log ..\<Script folder>\codegen to help identify if the candidates found are real correlations.
Extractors and appliers
DevWeb engine correlations are based on extractors and appliers. The extractors can be added automatically during code generation based on rule-based or record-based scans; or added manually to the script.
When scanning server responses, the Offline Script Generator adds an extractor for each identified dynamic value. This enables the value to be extracted and saved in a parameter. Then, the script code is scanned for the value, and matches are replaced with a reference to the parameter.
Tip: When the log level (in rts.yml) is set to debug or trace, the extractor substitutions are reported in the log. The log messages document each place in the script that the extractor is applied, and the substitute value used for each instance.
-
The extractors locate and extract the dynamic data in the responses. They can be based on boundary definitions or regular expressions, or can be used to query for object values inside JSON, XML, and HTML-formatted data. They also include converters for Base64, URL, and HTML encoding and decoding.
Extractors can also be added to your scripts to validate and manipulate content. For example, you can add a text check extractor using the failOn value option.
The available extractors include the following:
- BoundaryExtractor(name, options) : ExtractorObject
- RegexpExtractor(name, options) : ExtractorObject
- JsonPathExtractor(name, options) : ExtractorObject
- XpathExtractor(name, options) : ExtractorObject
- TextCheckExtractor(name, options) : ExtractorObject
- HtmlExtractor(name, options) : ExtractorObject
- CookieExtractor(name, options) : ExtractorObject
-
The appliers apply the extracted values, so that the saved parameter is used instead of the recorded value in subsequent requests to the server. You use the appliers to refine the way the extracted value is applied in the request. You can configure one or more appliers for each rule. See Extractors and appliers, below.
If no appliers are defined for a rule, a default applier is used.
The following table lists the applier types available.
The definitions for the supported parameters are provided here: Applier parameters
Applier type | Description | Supported parameters |
---|---|---|
simple | Replaces the extracted value with a dynamic parameter that was created from the extractor. The replacement can occur in any subsequent requests to the original response (the server response containing the extractor). |
|
boundary | Replaces an extracted value between defined boundaries (left and right), with a dynamic parameter that was created from the extractor. The replacement can occur in any subsequent requests to the original response (the server response containing the extractor). |
|
regexp | Replaces an extracted value that follows a regular expression, with a dynamic parameter that was created from the extractor. The replacement can occur in any subsequent requests to the original response (the server response containing the extractor). |
|
none |
Used when you want to enable the generation of extractors in the script, but do not want the extracted values to be applied in the script. This avoids the default behavior: If DevWeb cannot find where to apply the extracted values, then the extractor is not generated. |
n/a |
The following gives the definitions for the applier parameters. To see which parameters are supported for each applier type, see Applier types, above.
expression | string | The regular expression. |
flags | string | The flags for the regular expression. |
leftBoundary | string | The left boundary. |
rightBoundary | string | The right boundary. |
caseInsensitive | boolean |
Search boundaries using non-case sensitive comparison. Default: true |
occurrence | string/number |
The occurrence of the value to replace:
If occurrence is not defined, the first occurrence is replaced. |
scope |
string |
The relevant HTTP part in which the value will be replaced: url, querystring, headers, body, or all Default: all |
fieldName | list of strings |
A key (or keys) defining the focus within the scope area. For example, if the value for scope is |
Using correlation rules
Rules-based correlation scans use the rules defined in correlation rule definition files, <correlation_rules>.yml. You create customized correlation rule files, to add and edit rules for your specific DevWeb scripts.
Correlation rule files are, by default, located in the <DevWeb root folder>\CorrelationRules\ folder. This folder includes a predefined correlation rules file for an ASP.NET testing environment, correlation_ASP_NET.yml. You can edit this file, or make copies to use as templates. Edit and extend the rule definitions as needed for your testing environment. You can also refer to the Example file below.
Each correlation rule group includes Extractors and appliers. Within the file you can enable or deactivate each rule group, and enable or deactivate the whole correlation rules file. If rules scan is enabled in generator_config.yml, then during code generation the Offline Script Generator scans all *.yml files in the CorrelationRules folder, and loads all enabled rules.
Tip: Check out the video: Tutorial #7: Correlations.
To use correlation rules:
-
Define correlation rule files (*.yml) for your testing environment. Save the files in the default <DevWeb root folder>\CorrelationRules\ folder, or in a different location.
- Before recording your script, make sure that you have enabled all required correlation files and rule groups in the correlation *.yml files.
- Configure correlation parameters in the <DevWeb root folder>\generator_config.yml file, including setting rulesScanEnabled to
true
, and modifying the rulesDirectory path if you are using a different folder for the correlation rule files. (See Rule-based correlation, above.) - Record the script. When a match is found for a rule, an extractor for the value is added to the script.
The following gives a example of an annotated <correlation_rules>.yml file:
categories: - name: category1 #The name of the category, not used in code, this is for the user enabled: true #Allows the user to enable or disable all the rules in a category rules: #A list of rules - name: myRule1 #The name of the rule, used in the code the user writes to identify the rule. Must be unique across all categories type: boundary #The type of rule. Can be "boundary", "regexp", "json", "xpath" enabled: true #Allows the user to enable and disable particular rules options: #The options particular to this rule type leftBoundary: abc #Left boundary of the boundary-based correlation rightBoundary: efg #Right boundary of the boundary-based correlation caseInsensitive: false #Search boundaries using case-insensitive comparison (default: true) scope: all #Scope define the required HTTP part: headers, body, or all Default: all appliers: - type: simple #The type of the applier. One of "simple", "boundary", "regexp" or "none" which prevent apply automatically scope: headers #Scope define the required HTTP part: url, querystring, headers, body, or all Default: all options: occurrence: first #Occurrence focus on how many items to apply on all, first, last or valid number fieldName: ["cookie"] #Field Name allow the applier to focus on one or more key entries - name: myRule2 type: regexp enabled: true options: expression: "[0..9]*" #The regular expression. See https://github.com/google/re2/wiki/Syntax for details flags: "im" #The flags of the regular expression groupNumber: 0 #The group number (Default: 1 - for the first group) scope: body #Scope define the required HTTP part: headers, body, or all Default: all appliers: - type: boundary scope: url options: leftBoundary: le> #Left boundary of the boundary based correlation rightBoundary: </t #Right boundary of the boundary based correlation caseInsensitive: false #Search boundaries using case-insensitive comparison (default: true) - name: category2 enabled: true rules: - name: myRule3 type: json enabled: true options: path: abc #The JSONPath. See http://goessner.net/articles/JsonPath/ for details appliers: - type: regexp scope: all options: expression: "\\w+(.*?)\\w+" #The regular expression see https://github.com/google/re2/wiki/Syntax for details flags: "i" #The flags of the regular expression, please note the group number is default 1 no need to define it occurrence: first - name: anotherRule3 #Appliers section is optional on this case it will use the Default Applier type: xpath enabled: true options: path: //myEntry #The XPath, see https://godoc.org/gopkg.in/xmlpath.v2 for details
If no appliers are defined for a rule, the following default applier are used:
appliers: - type: simple scope: all options: occurrence: all
See also: