Synchronization and mapping scripts
This topic describes the batch scripts that allow you to automate your mappings and synchronizations.
CopySyncSets script
The CopySyncSets script allow you to automate the creation of type-specific project pairs for mapped types. Use this script to create a copy of the syncsets spanning the mapped types and introduce new project pairs. This script helps you avoid mapping errors that are common with manual mapping.
Before you run this script:
-
Create a property bag called copysyncsets.property.bag.txt in the usermaps folder.
-
Make sure you have a connection with at least one project mapped pair. The mapped pair may include several types, where each type has its own properties, such as calculated values, sync criteria, and server filters.
-
Stop the Micro Focus Connect service. If you do not stop the connection, Micro Focus Connect will automatically suspend it during the running of the script.
To copy the syncsets, use this script in one of the following modes:
-
Multiple targets to a single source project or workspace.
-
Multiple source projects to a single target project or workspace.
-
Multiple source projects mapped to multiple target projects.
Multiple targets to a single source
This mode copies multiple target projects to a single source project/workspace. For this mode, add the following keys to the property bag:
connection={ConnectionName}
templatetargetproject={MappedTargetProject}
templatetargetkey={MappedTargetProjectKey} (For Jira only)
templatesourceproject={MappedSourceProject}
targetproject={NewTargetProjectNameToBeMapped}
targetkey={NewTargetProjectKeyToBeMapped}
product=jira {For Jira only}
Multiple source projects to a single target
This mode copies multiple source projects to a single target project/workspace. For this mode, add the following keys to the property bag:
connection={ConnectionName}
templatetargetproject={MappedTargetProject}
templatetargetkey={MappedTargetProjectKey} (for Jira only)
templatesourceproject={MappedSourceProject}
sourceproject={NewSourceProjectNameToBeMapped}
product=jira (for Jira only)
Multiple source projects mapped to multiple target projects
This mode combines the capabilities of the two other modes, by allowing you to copy multiple source projects to multiple target projects. For this mode, add the following keys to the property bag:
connection={ConnectionName}
templatetargetproject={MappedTargetProject}
templatetargetkey={MappedTargetProjectKey} (For Jira only)
templatesourceproject={MappedSourceProject}
targetproject={NewTargetProjectNameToBeMapped}
targetkey={NewTargetProjectKeyToBeMapped}
sourceproject={NewSourceProjectNameToBeMapped}
product=jira (For Jira only)
Running this script creates new syncsets. In these syncsets, all the types in the connection's template projects that were selected, are mapped to the newly targeted projects within the connection. They include all of the sync criteria, calculated values, and server filters.
After running the scripts, restart the Micro Focus Connect service review the mappings and make sure that the script created them as required.
You can run this script as often as required in a sequence, for different datasets.
DataSourceUsers and UserMaps scripts
The DataSourceUsers and UserMaps scripts can be used together to automate the process of storing mapped user representations from different systems into Micro Focus Connect.
- DataSourceUsers: Extracts all users directly from the data sources.
- UserMaps: Pairs and imports users into the Micro Focus Connect database.
It is useful when the user data is accessible from a common authentication store such as Active Directory or LDAP.
To instruct Micro Focus Connect to automatically create user maps, see Automatic user mapping.
![Closed](../../Skins/Default/Stylesheets/Images/transparent.gif)
The following example shows the creation of a user map with the default user for a synchronization between Jira and ALM Octane.
This example creates a mapping from the Jira Display Name to the ALM OctaneEmail. Jira is currently restricted to display name. ALM Octane user representation can be set to either email, full name, or name
To create the user map:
- Determine the fields you want to map. Jira is currently restricted to display name. ALM Octane user representation can be set to either email, full name, or name.This example creates a mapping between the Jira display name and ALM Octane email.
- Exclude any data sources that are not required: Create a file named excludedatasources.txt in the ..\Connect\UserMaps folder and add all the data source names that you want to exclude. Separate the entries with a line break. Note: The data sources names must match exactly, including upper and lower case.
-
Perform an initial extraction of the user data:
- Open a command prompt as an administrator.
- Navigate to the Micro Focus ConnectUtilities folder. For details, see Batch utility scripts.
-
Run the DataSourceUsers script. This extracts all of the user data from the data sources to files. These files are given the names of the data sources with no extensions, in this example OCT_TEST and Jira Cloud. This step uses a unique key that exists in both data source files. In this example, the email address is unique in both systems and it is used as the key.
OCT_TEST file content
id|name|email|first_name|last_name|full_name|phone1|
5003|TomJones|tom.jones@opentext.com|Tom|Jones|Tom Jones|null|Jira Cloud file content
key|name|displayName|emailAddress|accountId
null|null|Tom Jones|C|6086f22fbb0b0500704ea7c3
-
Configure the fields used for the mapping by creating property bags for each data source. In this example, the data sources are named OCT_TEST and Jira Cloud, resulting in the following files: OCT_TEST.property.bag.txt and Jira Cloud.property.bag.txt. The names are case-sensitive.
-
The OCT_TEST.property.bag.txt has the following columns defined:
columns=email,email
-
The Jira Cloud.property.bag.txt has the following columns defined:
columns=emailAddress,displayName
-
-
Extract the user data again: Open a command prompt as an administrator, navigate to your Micro Focus ConnectUtilities folder, and run the DataSourceUsers script. This extracts all of the user data from the data sources to files, filling them with content:
-
The OCT_TEST file content:
tom.jones@opentext.com|tom.jones@opentext.com
-
The Jira Cloud file content:
tom.jones@opentext.com|Tom Jones
-
-
Create user maps:
- Open the mfcUserMaps.bat file in the Micro Focus ConnectUtilities folder in a text editor.
- Uncomment the line that has the pairs option by removing the REM from the start of the line. When run with the pairs signature it executes incremental inserts/updates of common missing keys and/or changed values for the file pairs.
-
Add the references to the data sources and a default user for each system, The user you designate as default, must exist.
"%JRE_PATH%\java.exe" -jar "%UTILITIES_PATH%\mfcUserMaps.jar" "%CONNECT_PATH%" "%USER_MAPS_PATH%" "|" "pairs" "%USER_MAPS_PATH%\OCT_TEST" "%USER_MAPS_PATH%\Jira Cloud" "OCT_TEST=tom.jones@opentext.com" "Jira Cloud=Tom Jones"
- Save the mfcUserMaps.bat file.
-
Open a command prompt as an administrator, navigate to your Micro Focus ConnectUtilities folder, and run the UserMaps script. This script stops the Micro Focus Connect Connect service, adds the user maps, and restarts the service. The results displayed in Micro Focus Connect are:
- The mapping key: TOM.JONES@OPENTEXT.COM
- User representation in Jira: Tom Jones
- User representation in ALM Octane: tom.jones@opentext.com
-
Automate the process: Since users can be added on both systems, it is a recommended practice to automate the process. Navigate to your Micro Focus ConnectUtilities folder, and edit the mfcPurge.bat file in a text editor. Add the bolded lines (the below code is based on the changes made to the batch files above):
"%JRE_PATH%\java.exe" -jar "%UTILITIES_PATH%\mfcAudit.jar" "%CONNECT_PATH%" "|" "%BACKUP_PATH%" "connectionnames=" "columnnames=" "datetime="
REM run the mfcDataSourceUsers.bat java command line text
"%JRE_PATH%\java.exe" -jar "%UTILITIES_PATH%\mfcDataSourceUsers.jar" "%CONNECT_PATH%" "%USER_MAPS_PATH%"
net stop ConnectWebServer
:loopStop
sc query ConnectWebServer | find "STOPPED"
if errorlevel 1 (
timeout 1
goto loopStop
)
REM run the mfcUserMaps.bat java command line text
"%JRE_PATH%\java.exe" -jar "%UTILITIES_PATH%\mfcUserMaps.jar" "%CONNECT_PATH%" "removeall"
"%JRE_PATH%\java.exe" -jar "%UTILITIES_PATH%\mfcUserMaps.jar" "%CONNECT_PATH%" "%USER_MAPS_PATH%" "|" "pairs" "%USER_MAPS_PATH%\OCT_TEST" "%USER_MAPS_PATH%\Jira Cloud" "OCT_TEST=tom.jones@opentext.com" "Jira Cloud=Tom Jones
…
If not already scheduled, add this script to your task scheduler to run on a daily basis. For details, see Purge files on Windows as a nightly task.
MatchByName script
Different products use different models for representing releases and sprints. For example:
- In Jira, sprints are independent of releases.
- In ALM Octane, sprints are children of releases.
- In QC/ALM, release cycles are children of releases, which are contained in release folders.
-
In Azure DevOps, there is no explicit concept of a sprint. The closest are iterations, which are described as tree structures.
The different models make it impractical to synchronize release and sprint entities in one product with their equivalents in the other product.
Instead, we recommend that users manually create releases and sprints (or their equivalents) in the synchronized projects. The release and sprint names in both projects should be identical, including their letter cases.
The MatchByName scripts build tables of matching IDs for the identical releases and sprints in the synchronized projects.
With the tables of matching IDs, Micro Focus Connect can synchronize field values in items, such as stories and defects, that reference releases and sprints.
The following scripts are available for mapping data types between selected pairs of products:
-
matchByNameAlmQcOctane
-
matchByNameAzureDevOpsAlmQc
-
matchByNameJiraAlmQc
-
matchByNameJiraOctane
The following tables describe how the scripts map the data types between pairs of synchronized products:
![Closed](../../Skins/Default/Stylesheets/Images/transparent.gif)
ALM/QC | ALM Octane |
---|---|
Releases | Releases |
ReleaseCycles | Sprints |
![Closed](../../Skins/Default/Stylesheets/Images/transparent.gif)
Azure DevOps data types | QC/ALM data types |
---|---|
Releases | Releases |
Iterations | ReleaseCycles |
![Closed](../../Skins/Default/Stylesheets/Images/transparent.gif)
Jira data types | ALM Octane data types |
---|---|
Releases | Releases |
Sprints | Sprints |
![Closed](../../Skins/Default/Stylesheets/Images/transparent.gif)
Jira data types | QC/ALM data types |
---|---|
Releases | Releases |
Sprints | ReleaseCycles |
Epics | RequirementFolders |
When run, the script explores the Micro Focus Connect database as follows:
- Discovers data sources based on the products specified in the script.
- Discovers the connections that use the data sources.
- Discovers the mapped source and target projects in the connections.
- Between the source and target projects, creates tables of matching IDs for identical pairs of releases and sprints (or their equivalents).
You can choose to run the scripts according to the following schedules:
- Nightly, as part of the Purge script, so that the Micro Focus Connect database can catch up with any newly-added releases or sprints (or their equivalents).
- On demand by the Micro Focus Connect administrator, based on advice from stakeholders, when new releases or sprints are added to any of the synchronized projects.
CustomSync
The CustomSync script reads cross references from the Micro Focus Connect database, and synchronizes them between the source and target endpoints. This script allows you to perform actions not available from the user interface, such as synchronizing multi-value sets to strings, and single or multi-value lists to similar lists, without having to perform value mapping.
The utility uses a structured property bag which describes the syntax for mapping target properties to master properties. It is supported for the following endpoints: ALM/QC, ALM Octane, Jira, Azure DevOps, Rally, VersionOne, GitLab, and ServiceNow.
Use the following syntax for the CustomSync script's property bag:
TargetType.TargetFieldName{,TargetFieldName,TargetFieldName,…}:SERVER=MasterType.MasterFieldName{,MasterFieldName,MasterFieldName,MasterFieldName,…};{MASTER|TARGET|BOTH}
…
{ConnectionName}={+|-}connectionname
{ConnectionName}={+|-}connectionname
…
{datasourcename}.defaultuser={userrepresentation}
…
allusers={true|false}
alwaysauthenticate={true|false}
resturls={true|false}
rawdata={true|false}
restheaders={true|false}
roles={true|false}
sqldml={true|false}
statistics={true|false}
usewatermarks={true|false}
Argument list
Argument | Meaning | Remarks |
---|---|---|
MasterType, TargetType | The type names | The JSON type names—not the label or display names. |
TargetFieldName, MasterFieldName | The field names | The JSON field names—not the label or display names. |
:SERVER | Generates Jira- specific JQL ignoring the projectkey from the Micro Focus Connect database syncset. | This optional property only applies to Jira endpoints. |
;MASTER|;TARGET|;BOTH |
The direction of synchronization: MASTER- a uni-directional synchronization from the target to master TARGET- a uni-directional synchronization from the master to target BOTH- a bi-directional synchronization.
|
For the BOTH option:
|
+connectionname, -connectionname |
The connections to include (+) or exclude (-) for this script. | The terms +connectionname and -connectionname are mutually exclusive; you cannot use them in the same property bag. Use +connectionname to specify all the connections you intend to include, or -connectionname to specify all the connections you want to exclude. |
{datasourcename}.defaultuser | The default user representation for user mapping for your data source. |
If you do not have a user map for the connection, add this entry. For example, for a Jira source, you could enter |
Property list
This section lists the optional properties.
Argument | Meaning | Remarks |
---|---|---|
allusers | Include all users | When set to true, the default value, all users are considered, When set to false, only users associated with participating projects in the connections/datasources defined in the CustomSync property bag will be considered. |
alwaysauthenticate | Send basic authentication credentials | When set to true, basic authentication credentials are sent with each REST request, even those that are not required by the Micro Focus Connect Jira data source. |
defaultlinktype | Specify a link type | If present, Micro Focus Connect uses the specified linktype name when creating Jira issue links, for example, defaultlinktype=Blocks . |
resturls, rawdata, and restheaders | Optional properties that generate files with REST URLs, raw data, or REST headers. |
These are optional properties primarily for debugging and analysis purposes. If you use resturls, an output file is generated. In the next run, Micro Focus Connect will archive it into a .zip file and back it up in the AppData/backup folder. Note: If you have a nightly purge scheduled, it deletes all files older than 10 days from the backup folder. |
roles | Use Jira roles | For Jira endpoints only: When set to true, only users with Jira roles will be considered. Roles can only be set to true if the allusers property is set to false. |
sqldml | SQL query logging | When set to true, this property logs each SQL query executed against the Micro Focus Connect database in preparation for the synchronization. |
statistics | Return statistics |
When set to true, this property returns a list of REST commands, and the number of times each of these commands was executed during the synchronization. It also provides the total time spent in sending and receiving data for the command. |
usewatermarks | Set a watermark |
When set to true, this property instructs the synchronization to set a watermark for the last run. This helps optimize the queries in the next run, as the synchronization is only done for the projects that changed. When set to false, the default value, the next synchronization will run against all projects in the syncsets. |
Follow these guidelines:
-
All contents of the property bag are case-sensitive.
-
The parameters of the property mappings must be in the order shown above.
-
Use a sequence of MasterFieldName parameters to synchronize a single target property to multiple master properties, such as MasterFieldName1,MasterFieldName2,MasterFieldName3
-
Use a sequence of TargetFieldName parameters to synchronize a single master property to multiple target properties, such as TargetFieldName1,TargetFieldName2,TargetFieldName3
-
Synchronizing multiple master properties to multiple target properties, such as TargetFieldName1,TargetFieldName2=MasterFieldName1,MasterFieldName2, is not supported.
-
This script synchronizes matched cross references per connection per syncset – source project, source type, target project, and target type. It does not create new artifacts at either endpoint, nor does it delete artifacts at either endpoint.
-
Properties specified in the custom sync property bag must not be included in the Micro Focus Connect core synchronization.
-
The CustomSync script does not use watermarks. Therefore for each run, it needs to query all artifact properties, types, and projects at each endpoint.
-
You can schedule the CustomSync script to run independently. If you have a scheduled purge using the Purge scripts, run this script as part of the purge, by adding it to the Purge script's .jar execution.
-
We recommend contacting the Support team at dl-connect-cfrd@microfocus.com when initially preparing this script.
Run the script
Run the CustomSync script in one of the following ways:
-
In the usermaps folder, add a shared property bag called customsync.property.bag.txt. Run the script, using the above syntax.
-
In the usermaps folder, specify individual, uniquely named property bags. Invoke the CustomSync script sync by passing the property bag name as the second parameter to the .jar in the script's.bat file. (The first parameter is always the path to the Micro Focus Connect installation folder.)
Example
The following is a sample property bag for an ALM Octane Jira synchronization, with a connection named Octane_JIRA_Demo. It synchronizes a multi-selectable Jira list to and from an ALM Octane multi-selectable custom list. It synchronizes:
-
an ALM Octane defect ID to a Jira custom string
-
a Jira key to an ALM Octane custom string
Bug.customfield_10235=defect.testcustomtoollist_udf;BOTH
Bug.custom_field_19834=defect.id;TARGET
Bug.key=defect.jira_id_udf;MASTER
You can add more connections using the syntax shown above, <my_connection_name>=+connectionname
See also: