Correlation Rules

In the context of this site, Correlation Rules represent a method of documenting those correlation rules that appear in a script as a web_reg_save_param_ex(...) function where, boundaries LB and RB are the prime criteria.

Without the DoxRunner process, correlation rules are added to a raw script manually. If the script needs to be re-recorded, this process is repeated, allbeit slightly faster as the scripter becomes more familiar with it.

The DoxRunner process requires correlation rules to be added to the Test Case or Solution document, not the script. They are applied to the script when the Process Raw operation is executed. If the script needs to be re-recorded, the effort to document them is significantly reduced - they only need to be reviewed and tailored if necessary - the Process Raw operation will deal with them each time it is executed.

The Process Raw operation makes heavy use of the CodeGenerationLog.txt file, so the script's ...\data\... folder must be in place. If it's not there it will warn you and if you proceed it will ignore the documented Correlation Rules.

As with all Managed Items, Correlation Rules are defined as rows within a Table that is located in the Correlation Rules section of a Test Case and/or the Solution document.

Getting Started

A Correlation Rules section exists in both the Solution document and the Test Case Template document when they are downloaded from the DoxRunner site. Because it exists in the Test Case Template document, all Test Cases created from it will also contain a Correlation Rules section.

When the solution document is downloaded, review the Correlation Rules section and update if necessary.
When the Test Case Template document is downloaded, review the Correlation Rules section and update if necessary.

Read and understand how Correlation Rules are documented as set out further down this page.

It is also useful to read and understand how the Process Raw operation uses the documented Correlation Rules to parameterise a raw script.

How it works - Summary

A summary of the process follows:

Create a Test Case;
Prepare a fresh recording using the DoxRunner way (this will be considered to be a raw script);
Manually identify all correlation rules;
Add them to the Correlation Rules section of the Test Case or the Solution document (refer to the Documentation section below);
Execute the Process Raw operation;
Check the updated script;
If necessary, update the documented Correlation Rules and re-execute the Process Raw operation;
Repeat until the script is fully operational.

Each Correlation Rule can be considered to come in pairs - one or more Request rules and one or more Response rules for the same Parameter Name.

Stage 1 of the DoxRunner Process Raw operation searches the action files for any occurrence of the Request rules. If found, it replaces the value with a temporary parameter. The name of this parameter is structured.

Example: {pSAML_54_NotFound}, where 54 represents a Unique Parameter ID created by DoxRunner.

Stage 3 of the DoxRunner Process Raw operation searches the ...\data\CodeGenerationLog.txt file for a response that matches the related Response rule. If found, all occurrences of the temporary parameter name are finalised.

For example {pSAML_54_NotFound} could be changed to {pSAML_t67_54}, where t67 represents the snapshot where the response was found, and 54 represents a Unique Parameter ID.

A web_reg_save_param_ex(..) function is inserted into the script immediately prior to the request that resulted in the required response.

Documentation

As with all Managed Items, Correlation Rules are documented in a DoxRunner section.

A Correlation Rules section may appear in three places:

The Test Case Template document;
Any Test Case;
The Solution document.

It is optional. Unless you have deleted it from the Test Case Template, it will be included in all new Test Cases you create.

It is recommended that the section be on a page that is portrait oriented. DoxRunner defaults to portrait.

Why document Correlation Rules in the Solution document? If you find yourself defining the same rule over and over in more than one Test Case, it may be worthwhile putting those in the Correlation Rules section that is included in the Solution document. The Process Raw operation will reference those first, then reference any that are specified in the Test Case. If the same rule exists in both, the one in the Test Case overrides the one in the Solution document.

Although the table in the Correlation Rules section is relatively complex, the structure of the section itself is relatively simple. The important components are identical to all other Managed Items that contain a table. That is, the structure of the section has the following components:

Section Title;
- Section Bookmark;
Section Description;
A Table;
- A Heading Row;
- A minimum of two Data Rows (unlike other Managed Items, where the minimum is one);
- A minimum of four Columns.

One Data Row must contain a Request Rule and one must contain a Response rule for the same Parameter Name.

Deleting or re-adding a Correlation Rules section in a Test Case is best done via the Test Cases Operations menu.

Deleting or re-adding a Correlation Rules section in the Test Case Template and Solution document can only be done manually.

Adding and deleting a rule in a Correlation Rules table can only be done manually.

Example

The example in the illustration below shows the definition of six Correlation Rules for two parameter names.

Correlation Rules Section

As listed above and shown in the illustration above, a Correlation Rules section is structured with the following components:

Section Title;
- Section Bookmark;
Section Description;
A Table;
- A Heading Row;
- A minimum of two Data Rows;
- A minimum of three Columns, but ideally four Columns.

The Section Bookmark:

Mandatory;
Location: Immediately before the Section Title;
Structure: Depends on which document it is embedded in:
- Test Case: P_PT001_CorrelationRules (where PT001 is the Test Case ID)
- Test Case Template: P_CorrelationRules
- Solution document: V_CorrelationRules

The Section Title:

Mandatory;
Location: Immediately after the Bookmark;
Structure: Free-format text;
Style: I_Heading n or I_Appendix n, where n can be an integer from 1 to 5;
Length: Cannot be longer than 200 characters.

The Body Text

Optional;
Location: Between the Section Title and the Table;
Structure: Free-format text;
Style: I_BodyText;
Length: Cannot be longer than 1,000 characters.

The Table:

Mandatory;
Location: Immediately below the Body Text (or Section Title if there is no Body Text):
A minimum of three rows:
- A Heading row and two Data rows (one data row for a Request, the other for its Response);
- If there are no rules, the last Data row can contain empty cells (or the section can be deleted);
A minimum of three columns:
- A Parameter Name column, a Role column, and a Configuration column;
- More columns can be added;
- A Description column is recommended.
All cells in the Parameter Name, Role, and Configuration columns must conform to specific rules, as described below.

Correlation Rules Column Headings

The four default columns within the table are described below. More columns can be inserted or appended if required.

Parameter Name (mandatory);
Role (mandatory);
Configuration (mandatory);
Description (optional).

Parameter Name

The documented parameter name is only the main part of the name that is actually used in the script. The final name is generated by the Process Raw operation. It will have a suffix as shown in the example below. Because of this, the length of the name in the Parameter Name column must take the length of this suffix into consideration.

Example using standard parameter names

In this example, the documented Parameter Name is: pThisIsMyParameterName.

If Stage 1 of the Process Raw operation identifies a value in the script that matches its Request rule, then it replaces the value in the Request with a temporary Parameter name structured as: pThisIsMyParameterName_132_NotFound
- where 132 is a Unique Parameter ID that DoxRunner creates;
If Stage 3 of the Process Raw operation identifies a Response that matches its corresponding Response rule, then the temporary Parameter name is finalized to: pThisIsMyParameterName_t54_132
- where t54 is the snapshot name of the request resulted in the response, and 132 is a Unique Parameter ID that DoxRunner created in Stage 1.

Example using encoded values

If DoxRunner has identified a Response that is an encoded version of the value in the Request, then Stage 3 of the Process Raw operation will insert a function that does the encoding. In this case the final Parameter Name will have an additional suffix appended.

The encoded Parameter that is used in the web_reg_save_param_ex(...) function will be:

pEncodedParameter_t54_132_Response

The function that does the encoding is located just prior to the first Request that uses the encoded value.

The example here uses function ConvertHexToASCII(...). It is structured like this:

iResult=ConvertHexToASCII("{pEncodedParameter_t54_132_Response}", "pEncodedParameter_t54_132_Request", "\\x");
if (iResult!=0)
lr_exit(LR_EXIT_ITERATION_AND_CONTINUE, LR_PASS);

In the example above, the final Parameter name that replaces the temporary Parameter Name in the Request will be:

pEncodedParameter_t54_132_Request

Parameter name not documented

If a rule is defined in the table, but the Parameter Name is missing , then the rule is ignored by the Process Raw operation.

Role

The Role is mandatory. Only the text Request or Response is allowed here.

There can be multiple Request and/or Response rules for the one Parameter Name.

The Role of Request is used in Stage 1 of the Process Raw operation. It looks for any value that meets the Request's Configuration and, if a suitable value is found, it is replaced by the temporary Parameter Name - one ending in _NotFound.

The Role of Response is used in Stage 3 of the Process Raw operation. It searches the ...\data\CodeGenerationLog.txt file for a response with text that matches the Response's Configuration and that found by the corresponding Request rule.

If the Request rule wasn't found in the script, its associated Response rule is ignored.

If the Response rule that matches the Request rule and its value wasn't found in the CodeGenerationLog.txt file, the request is still parameterised. Since the response could not be found, the snapshot filename cannot be appended to the parameter name, nor can the web_reg_save_param_ex(...) function's location be determined. So the parameter is assigned at the top of Action.c and formatted as follows: <ParameterName>_n_NotFound where n is the Unique Parameter ID.

Configuration

The Configuration cell for each rule consists of a Key Word followed by zero or more Qualifiers.

Key Words

Only one key word per Configuration. It should appear on the first line, followed by a colon and a space.

Boundary
XML
URL
Item Data
JSON

Qualifiers

The key word can be followed by zero or more qualifiers, each followed by a colon and a space.

Description

The Description is optional, and can be any text that adds value to the implementation of the rule.

Configuration Key Words

The Configuration cell for each rule consists of a Key Word followed by zero or more Qualifiers.

Key Words

One key word per Configuration. It should appear on the first line, followed by a colon.

Boundary
XML
URL
Item Data
JSON

Configuration Key Word: Boundary

The Boundary configuration mimics the standard web_reg_save_param_ex(...) requirements by specifying LB and RB qualifiers.

One LB and one RB qualifier is mandatory and are the minimum requirements. Only one LB per rule is allowed, but multiple RBs is supported.

The illustration below shows the simplest form. Note that the Key Word Boundary is missing - it is optional. DoxRunner infers Boundary from the presence of both LB and RB Qualifiers.

Refer to the illustration below for a view of the Boundary documentation and how the Process Raw operation uses it, along with the response in the CodeGeneraionLog.txt file, to correlate the Request and insert the web_reg_save_param_eEx(...) function prior to the Request that gives rise to the relevant Response (in this case the Request at Snapshot txxx).

Documenting a parameter requires two records - a Request rule and a Response rule. Although the example above shows both the Request and Response using the Boundary Key Word, it is not necessary for them to be the same. One could be XML and the other could be a Boundary rule or a URL rule, or any one of the other Key Words. Note that the backslashs in the Response are not escaped - DoxRunner will automatically escape them.

The Request Identification part of Stage 1 of the Process Raw operation identifies all Requests that match the rule.

When identified, Stage 1 replaces the value with a temporary parameter name with a Unique Parameter ID. The original value is appended as a comment. Example: {pSAPClient_23_NotFound} // pSAPClient_23_NotFound = 786 where 23 is the Unique Parameter ID and 786 is the Value.

This is an extract from the Response at snapshot t21 in the CodeGenerationLog.txt file that contains the required text.

The Response Correlation part of Stage 3 of the Process Raw operation identifies the Response that matches the Response rule.

The web_reg_save_param_ex(...) function is inserted immediately before the Request that resulted in the Response.

Escape characters are added as required.

If a Scope qualifier is not documented, the default of Scope=All is used.

All instances of the temporary parameter are changed to include the Snapshot name of the Response. The original value is appended as a comment. Example: {pSAPClient_t21_23} // pSAPClient_21_23 = 786 where t21 is the Snapshot name containing the required Response, 23 is the Unique Parameter ID, and 786 is the Value.

Configuration Key Word: XML

Syntax: XML: Tag Name

As the name suggests, a Configuration with Key Word XML specifies any Request or Response text that is formatted as XML.

In its simplest form an XML Configuration Item will consist of the Key Word 'XML' followed by a colon, a space, then the Tag Name.

The Process Raw operation uses it to correlate a Raw Script. It searches for the end tag and then processes the value. For example if the Key Word is "XML: BigOne", then it will search for the text "</BigOne>". When found, it identifies the Value as the text between the start tag and the end tag, and it updates the script accordingly. For example if the Key Word is "XML: BigOne" and the Request contains "<BigOne>Huge</BigOne>", then it will search the Responses for any text "Huge" that meets the corresponding Response rule for the same Parameter Name.

Note that it ignores elements formatted like this: "<BigOne/>".

The rule does not specify delimiters. Delimiters <, /, and > are the most common, however DoxRunner is not limited to these. It will explore those listed in the table below and update the script as appropriate:

Refer to the illustration below for a view of the XML documentation and how the Process Raw operation uses it, along with the Response in the CodeGeneraionLog.txt file, to correlate the Request and insert the web_reg_save_param_eEx(...) function prior to the Request that gives rise to the relevant Response (in this case the Request at Snapshot t8).

Documenting a Correlation parameter requires two records - a Request rule and a Response rule. Although the example above shows both the Request and Response using the XML Key Word, it is not necessary for them to be the same. One could be XML and the other could a Key Word of Boundary or URL, or any other. The XML Name in the example above is _dOffset, while that in the Response is Dual Offset.

The Request Identification part of Stage 1 of the Process Raw operation identifies all Requests that match the rule.

When identified, Stage 1 replaces the value with a temporary parameter name that includes a Unique Parameter ID. The original value is appended as a comment. Example: {pBreak_10_NotFound} // pBreak_t4_10 = 20140588_10:00 where 10 is the Unique Parameter ID and 20140588_10:00 is the Value. Note that the XML delimiters found by DoxRunner are the common ones (/, >, and /) as shown at row XML1 in the XML Table above. DoxRunner is not limited to these. If it cannot find any in the XML1 row of the table, it will look for those in the XML2 row, and so on.

The Response Correlation part of Stage 3 of the Process Raw operation identifies the Response that matches the Response rule, updates the temporary Request, and Creates the web_reg_save_param_ex(...) function.

This is an extract from the Response at snapshot t8 in the CodeGenerationLog.txt file that contains the required text. Note that the XML delimiters found by DoxRunner are those at XML4 in the XML Table above (~0060, ~0062, and ~0047). That is, DoxRunner has searched the Response for the delimiters from rows XML1 through XML3 and was only successful when those at row XML4 were tried. Note also that only the right delimiter is used in the LB. This is in case any Attributes are present and may be variable, thereby negating the capture of the Response value. There is one Attribute in the example and it is "Large=True".

The web_reg_save_param_ex(...) function is inserted immediately before the Request that resulted in the Response. Escape characters are added as required. If a Scope qualifier is not documented, the default of Scope=All is used.

All instances of the temporary parameter are changed to include the Snapshot name of the Response. The original value is appended as a comment. Example: {pSAPClient_t21_23} // pSAPClient_21_23 = 786 where t21 is the Snapshot name containing the required Response, 23 is the Unique Parameter ID, and 786 is the Value.

If you were to use the Boundary Key Word instead of the XML Key Word, then the documentation for the above example would look like the illustration below, which requires more typing.

Configuration Key Word: URL

Syntax: URL: URL Name

As the name suggests, a Configuration with Key Word URL specifies any Request or Response text that is formatted as URL.

In its simplest form a URL Configuration Item will consist of the Key Word 'URL' followed by a colon, a space, then the URL Name.

Delimiters will be these characters: ? ; & "

For example if the Key Word is "URL: BigOne", then it will search for the text one of these:

?BigOne=
;BigOne=
&Bigone=

Assuming it has found &BigOne=, it will then search for one of these delimiters:

&
?
;
"

If the line looks like this: "...?language=EN&BigOne=Huge&theme=gold" then DoxRunner will select & as the trailing character, not the quote, because it is closer to &BigOne=. Therefore the Value to correlate will be Huge.

The rule does not specify delimiters. The 4 delimiters listed above are the most common, however DoxRunner is not limited to these. It will explore those listed in the table below and update the script as appropriate:

Configuration Qualifiers

The Configuration cell for each rule consists of a Key Word followed by zero or more Qualifiers.

Qualifiers

The key word can be followed by zero or more qualifiers, each followed by a colon and a space.

Configuration Qualifier: LB

Syntax:

LB: Left Buffer

If an LB Qualifier is detected in a Configuration cell, the Key Word Boundary is assumed.

If it's not followed by one or more RB Qualifiers, then the rule is ignored.

The Left Buffer follows the rules for the LB in a web_reg_save_param_ex(...) function with one exception: Escape characters are not to be included - DoxRunner will insert them automatically. Just use the value from the CodeGenerationLog.txt file verbatim - this saves time.

Configuration Qualifier: RB

Syntax:

RB: Right Buffer

There can be more than one RB Qualifier, each on a separate line. They all must be preceded by an LB Qualifier, which DoxRunner infers the Key Word Boundary..

If it does not follow an LB Qualifiers, then the rule is ignored.

The Right Buffer follows the rules for the RB in a web_reg_save_param_ex(...) function with one exception: Escape characters are not to be included - DoxRunner will insert them automatically. Just use the value from the CodeGenerationLog.txt file verbatim - this saves time.

Configuration Qualifier: Replace Option

Syntax:

Replace Option: Option

Options:

"All Values" and "By Rule".

The default is to select a Request Value "By Rule".

The purpose of the Replace Option is to save time documenting Request rules. If a correlated value is quite long, and obviously unique, then multiple Request rules need not be documented - only the first one needs to be. From then on, if DoxRunner encounters that value, it will automatically correlate it.

It is only valid for a Request Rule. Also, it should only be used when it is certain that any value selected by the Request rule is unique throughout the script.

When Stage 1 of the Process Raw operation identifies a value, it parameterises it with a temporary parameter name. If another rule identifies another instance of that value, it parameterises it with a new temporary parameter name.

If that value is unique, then the second rule and its temporary parameter is wasteful.

For example, values such as Relay State, ViewState, SAMLRequest, Salt, Cache ID, are pretty much guaranteed to be unique. So only one rule is necessary. All subsequent instances of the value can be automatically parameterised with the same temporary parameter.

This reduces the amount of documentation needed.

In general, the shorter the value text, the less likely it'll be unique. The experience of the scripter is relied upon to determine the uniqueness of the value, and therefore the application of the "All Values" setting for the Replace Option qualification.

Configuration Qualifier: Scope

Syntax:

Scope: Option

Options:

"All", "Headers", "Body", "Cookies".

The Scope qualifier is used to determine how to configure the SEARCH_FILTERS argument in the web_reg_save_param_ex(...) function.

It is optional. If not included, it defaults to "Scope=All".

Configuration Qualifier: Constraint

The Replace Option is only valid for a Request Role.

The Process Raw operation determines the request value as specified by the Signature, LB, and RB, then it parameterises the values in all subsequent requests without regard to the Signature, LB, and RB. This means that you need to be very sure that the value will be unique. For example, ViewState, SAMLRequest, Salt, Cache ID, are pretty much guaranteed to be unique, so the Value rule can be applied. In general, the shorter the value text, the less likely it'll be unique. The experience of the scripter is relied upon to determine the uniqueness of the value.

Configuration Qualifier: Length

Syntax:

Length: Integer

Length: = Integer

Length: > Integer

Length: < Integer

Length: >= Integer

Length: <= Integer

The Length Qualifier limits the selection of a Request or Response Value to the number of characters specified.

The Process Raw operation determines the request value as specified by the Signature, LB, and RB, then it parameterises the values in all subsequent requests without regard to the Signature, LB, and RB. This means that you need to be very sure that the value will be unique. For example, ViewState, SAMLRequest, Salt, Cache ID, are pretty much guaranteed to be unique, so the Value rule can be applied. In general, the shorter the value text, the less likely it'll be unique. The experience of the scripter is relied upon to determine the uniqueness of the value.

Configuration Qualifier: Value

The Replace Option is only valid for a Request Role.

The Process Raw operation determines the request value as specified by the Signature, LB, and RB, then it parameterises the values in all subsequent requests without regard to the Signature, LB, and RB. This means that you need to be very sure that the value will be unique. For example, ViewState, SAMLRequest, Salt, Cache ID, are pretty much guaranteed to be unique, so the Value rule can be applied. In general, the shorter the value text, the less likely it'll be unique. The experience of the scripter is relied upon to determine the uniqueness of the value.

Reverse Engineer

Not implemented yet.

It's quite tedious to manually extract correlation rules from an existing script and place them into the table of the Correlation Rules section.

The Reverse Engineer operation has a function that will extract the Response rules from an existing script and place them into the table. It will also create a corresponding Request rule, but that will only be partially populated. It can't fully populate the Request rule because it's too difficult to programatically determine the best LB and RB - this is better done manually.

Process Raw Operation

The Process Raw operation executes in three stages. Only Stages 1 and Stage 3 are relevant for Correlation Rules.

.

Unique Parameter ID

Often the same Parameter Name appears in a Request multiple times and/or in a Response, but with different rules. The Unique Parameter ID ensures that they are implemented separately.

The solution that DoxRunner implements is to assign a unique number to all parameter names that arise from Correlation Rules.