Batch utility scripts

The batch utility scripts allow you to use the Micro Focus Connect capabilities more efficiently.

In this topic:

• General run notes
• ChangeDataSource
• ClearWaterMarks
• CopySyncSets
• Cross Reference scripts
• CustomSync
• DataSourceUsers and UserMaps
• DataSourceExtracts
• Fix
• Maintenance
• MatchByName
• Connection scripts
• Purge scripts
• Audit script
• QueryTemplates
• Rename
• ResetPassword

General run notes

The following notes apply to all batch scripts:

• Each batch file should be run as Administrator.

• The complete syntax for a batch script is mfc<script_name>[.bat|.sh], depending on your operating system. For example, to run the Purge script on Windows, use mfcPurge.bat in your syntax. For simplicity, this documentation only refers to the script name, without the mfc prefix and without the extension.

• If the script contains a placeholder value, for example, ConnectionName, you must edit it in a text editor and replace the placeholder value with your own value. Make sure to use the exact case-sensitive names for your values.

• To retrieve a list of your connection names, run mfcQueryStatus.bat. From the generated list, copy and paste the exact name of the connection into your script.

• The scripts are located in the utilities folder under the Micro Focus Connect installation.

• The scripts modify the Micro Focus Connect database.

• For most of the scripts, you should stop the Micro Focus Connect server, run the appropriate utility, and then restart the Micro Focus Connect server. For details, see Connection scripts.

• The following scripts can run without stopping the Micro Focus Connect server: DataSourceUsers, QueryTemplates, RunOneIteration, StartConnection, and Audit.

Back to top

ChangeDataSource

The ChangeDataSource script changes the master or target data source of an existing connection.

The original and replacement data sources must represent the same tool. Meaning, you can replace Jira for Jira and ALM Octane for ALM Octane. The two data sources must also target the same endpoint URL.

The script includes three variables: ConnectionName, OldDataSourceName, NewDataSourceName

Replace the values with the exact, case-sensitive names of the entries, as displayed in the Micro Focus Connect UI.

Back to top

ClearWaterMarks

The ClearWaterMarks script resets the watermarks for all active connections.

Resetting watermarks forces a full synchronization by each connection on the first iteration after the server restarts.

You can add parameters to the script to achieve the following results:

Parameter	Result
removenulls	Removes from the Micro Focus Connect database all cross references whose source or target value is null.
removemismatches	Removes all matched pair cross references, where either the source or the target value is in the Micro Focus Connect database but not in the corresponding endpoint product. The script targets the following endpoints only: ALM Octane, ALM/QC, Jira, Azure DevOps, Version One, Rally, or ServiceNow.
(no parameter)	Clears all the watermarks for all connections.

Back to top

CopySyncSets

The CopySyncSets script allow you to automate the creation of type-specific project pairs for mapped types. Use this script to create a copy of the syncsets spanning the mapped types and introduce new project pairs. This script helps you avoid mapping errors that are common with manual mapping.

Before you run this script:

1. Create a property bag called copysyncsets.property.bag.txt in the usermaps folder.

2. Make sure you have a connection with at least one project mapped pair. The mapped pair may include several types, where each type has its own properties, such as calculated values, sync criteria, and server filters.

3. Stop the Micro Focus Connect service. If you do not stop the connection, Micro Focus Connect will automatically suspend it during the running of the script.

To copy the syncsets, use this script in one of the following modes:

• Multiple targets to a single source project or workspace.

• Multiple source projects to a single target project or workspace.

• Multiple source projects mapped to multiple target projects.

Multiple targets to a single source

This mode copies multiple target projects to a single source project/workspace. For this mode, add the following keys to the property bag:

Multiple source projects to a single target

This mode copies multiple source projects to a single target project/workspace. For this mode, add the following keys to the property bag:

Multiple source projects mapped to multiple target projects

This mode combines the capabilities of the two other modes, by allowing you to copy multiple source projects to multiple target projects. For this mode, add the following keys to the property bag:

Running this script creates new syncsets. In these syncsets, all the types in the connection's template projects that were selected, are mapped to the newly targeted projects within the connection. They include all of the sync criteria, calculated values, and server filters.

After running the scripts, restart the Micro Focus Connect service review the mappings and make sure that the script created them as required.

You can run this script as often as required in a sequence, for different datasets.

Back to top

Cross Reference scripts

The following utilities circumvent the need to export the cross references template of the target connection, manually set up cross references in the template, and import them one syncset at a time. When working with a large number of syncsets, manual exports and configurations are prone to errors.

CopyXrefs

The CopyXrefs script copies cross references from a common projects-based connection to a corresponding type-specific projects connection. For details on common projects and type-specific projects, see Add projects and rules to a connection.

The script includes two variables: SOURCE_CONNECTION and TARGET_CONNECTION.

Replace the values with the exact, case-sensitive names of the corresponding connection names, as displayed in the Micro Focus Connect UI.

ImportXrefs

The mfcImportXRefs script imports all cross references from an exported cross reference file to an identical connection. It imports the cross references across all types and project pairs described by the exported XML file.

The script includes two variables: CROSS_REFERENCE_XML_FILE_NAME , in the usermaps folder, and TARGET_CONNECTION.

Replace the values with the exact, case-sensitive names of the corresponding cross reference XML file and connection name, as they appear in the Micro Focus Connect user interface.

A common use case for this utility is copying a connection from one Micro Focus Connect instance to another.

To copy a connection:

1. Perform an export of the first Micro Focus Connect instance to generate a Connection.xml. For details see Import, export, and back up data.

2. Export the cross references from the first Micro Focus Connect instance.

3. Import the connection file Connection.xml into the second Micro Focus Connect instance. You may want to rename the XML file before you import it, to allow you to differentiate between the XML files.

4. Copy the cross references XML file to the machine running the second Micro Focus Connect instance.

5. Import the cross references using mfcImportXRefs.bat|sh.

Back to top

CustomSync

The CustomSync script reads cross references from the Micro Focus Connect database, and synchronizes them between the source and target endpoints. This script allows you to perform actions not available from the user interface, such as synchronizing multi-value sets to strings, and single or multi-value lists to similar lists, without having to perform value mapping.

The utility uses a structured property bag which describes the syntax for mapping target properties to master properties. It is supported for the following endpoints: ALM/QC, ALM Octane, Jira, Azure DevOps, Rally, VersionOne, and ServiceNow.

Use the following syntax for the CustomSync script's property bag:

Argument list

Argument	Meaning	Remarks
MasterType, TargetType	The type names	The JSON type names—not the label or display names.
TargetFieldName, MasterFieldName	The field names	The JSON field names—not the label or display names.
:SERVER	Generates Jira- specific JQL ignoring the projectkey from the Micro Focus Connect database syncset.	This optional property only applies to Jira endpoints.
;MASTER|;TARGET|;BOTH	The direction of synchronization: MASTER- a uni-directional synchronization from the target to master TARGET- a uni-directional synchronization from the master to target BOTH- a bi-directional synchronization.	For the BOTH option: The synchronization runs slower, since history queries need to be executed for each endpoint artifact. Only supported for ALM/QC, ALM Octane, Jira, Azure DevOps, and ServiceNow.
+connectionname, -connectionname	The connections to include (+) or exclude (-) for this script.	The terms +connectionname and -connectionname are mutually exclusive; you cannot use them in the same property bag. Use +connectionname to specify all the connections you intend to include, or -connectionname to specify all the connections you want to exclude.
{datasourcename}.defaultuser	The default user representation for user mapping for your data source.	If you do not have a user map for the connection, add this entry. For example, for a Jira source, you could enter MyDataSource.defaultuser=Tom Jones

Property list

This section lists the optional properties.

Argument	Meaning	Remarks
allusers	Include all users	When set to true, the default value, all users are considered, When set to false, only users associated with participating projects in the connections/datasources defined in the CustomSync property bag will be considered.
alwaysauthenticate	Send basic authentication credentials	When set to true, basic authentication credentials are sent with each REST request, even those that are not required by the Micro Focus Connect Jira data source.
defaultlinktype	Specify a link type	If present, Micro Focus Connect uses the specified linktype name when creating Jira issue links, for example, defaultlinktype=Blocks.
resturls, rawdata, and restheaders	Optional properties that generate files with REST URLs, raw data, or REST headers.	These are optional properties primarily for debugging and analysis purposes. If you use resturls, an output file is generated. In the next run, Micro Focus Connect will archive it into a .zip file and back it up in the AppData/backup folder. Note: If you have a nightly purge scheduled, it deletes all files older than 10 days from the backup folder.
roles	Use Jira roles	For Jira endpoints only: When set to true, only users with Jira roles will be considered. Roles can only be set to true if the allusers property is set to false.
sqldml	SQL query logging	When set to true, this property logs each SQL query executed against the Micro Focus Connect database in preparation for the synchronization.
statistics	Return statistics	When set to true, this property returns a list of REST commands, and the number of times each of these commands was executed during the synchronization. It also provides the total time spent in sending and receiving data for the command.
usewatermarks	Set a watermark	When set to true, this property instructs the synchronization to set a watermark for the last run. This helps optimize the queries in the next run, as the synchronization is only done for the projects that changed. When set to false, the default value, the next synchronization will run against all projects in the syncsets.

Follow these guidelines:

• All contents of the property bag are case-sensitive.

• The parameters of the property mappings must be in the order shown above.

• Use a sequence of MasterFieldName parameters to synchronize a single target property to multiple master properties, such as MasterFieldName1,MasterFieldName2,MasterFieldName3

• Use a sequence of TargetFieldName parameters to synchronize a single master property to multiple target properties, such as TargetFieldName1,TargetFieldName2,TargetFieldName3

• Synchronizing multiple master properties to multiple target properties, such as TargetFieldName1,TargetFieldName2=MasterFieldName1,MasterFieldName2, is not supported.

• This script synchronizes matched cross references per connection per syncset – source project, source type, target project, and target type. It does not create new artifacts at either endpoint, nor does it delete artifacts at either endpoint.

• Properties specified in the custom sync property bag must not be included in the Micro Focus Connect core synchronization.

• The CustomSync script does not use watermarks. Therefore for each run, it needs to query all artifact properties, types, and projects at each endpoint.

• You can schedule the CustomSync script to run independently. If you have a scheduled purge using the Purge scripts, run this script as part of the purge, by adding it to the Purge script's .jar execution.

• We recommend contacting the Support team at dl-connect-cfrd@microfocus.com when initially preparing this script.

Run the script

Run the CustomSync script in one of the following ways:

• In the usermaps folder, add a shared property bag called customsync.property.bag.txt. Run the script, using the above syntax.

• In the usermaps folder, specify individual, uniquely named property bags. Invoke the CustomSync script sync by passing the property bag name as the second parameter to the .jar in the scriipt's.bat file. (The first parameter is always the path to the Micro Focus Connect installation folder.)

Example

The following is a sample property bag for a ALM Octane Jira synchronization, with a connection named Octane_JIRA_Demo. It synchronizes a multi-selectable Jira list to and from an ALM Octane multi-selectable custom list. It synchronizes:

• an ALM Octane defect ID to a Jira custom string.

• a Jira key to an ALM Octane custom string.

You can add more connections using the syntax shown above, <my_connection_name>=+connectionname

Back to top

DataSourceUsers and UserMaps

The DataSourceUsers and UserMaps scripts can be used together to automate the process of storing mapped user representations from different systems into Micro Focus Connect.

• DataSourceUsers: Extracts all users directly from the data sources.
• UserMaps: Pairs and imports users into the Micro Focus Connect database.

It is useful when the user data is accessible from a common authentication store such as Active Directory or LDAP.

To instruct Micro Focus Connect to automatically create user maps, see Automatic user mapping.

Back to top

DataSourceExtracts

The DataSourceExtracts script extracts a data dump of any type, any subset of fields, for any project across seven products: QC/ALM, ALM Octane, Jira, Azure DevOps, Rally, VersionOne, and ServiceNow.

The command syntax is:

Parameter	Details
"almqc"	Can be one of the following: almqc, octane, jira, azdo, rally, versionone, or servicenow.
"..\pathTo\almqc.property.bag.txt"	The path to the property bag for the product described by the first parameter.

You can run the DataSourceExtracts script on any machine that has the java jre installed, not just on the Micro Focus Connect server.

The utility receives as input a property bag that describes the connectivity and credentials for the target product.

The parameters in the property bag are a set of key-value pairs in the form of key=value

Key and value guidelines

• All the keys must be lowercase.
• Spaces, tabs or special characters are not supported in the key.
• Do not use quotation marks for the values.
• Value strings are case sensitive, and must match the corresponding data elements in the targeted product.

Property bag structure per product

Following are structures of each property bag with the minimum required parameters:

Proxy servers

If your endpoint system requires proxy servers, you need to specify their keys in the property bag. Use the following keys:

For details, see Proxy setup.

Optional parameters

Following are parameters that you can append to the command:

Parameter	Description
attributeids	True: Returns the underlying id’s of related fields. False: Returns the names of related fields. Default: true
columns	An array of field names. When specified, restricts the output data table to the set of specified fields. If not specified, all the fields of the type are returned. The parameter also directs the engine to optimize the query and fetch all the data in a single round trip. columns=columnname1,columnname2,columnname3,…
fetchusers	When set to false, no users are queried from any participating endpoint systems. Note: Use this property with caution, since when enabled, it may cause a conflict in cases where user fields are required in the synchronization. Default: true
fieldmetadata	Queries and writes out the list of all fields for the type specified in the property bag. Default: false
fieldseparator	Splits the output row based on the specified characters. If not specified, the default is | fieldseparator=^$# splits each field value in a row out separated by the characters specified in the pattern.
headers	Directs the engine to append the field/property/column name to each data value in the output Default: true
logrestheaders	Appends the rest http send/receive headers for each executed rest url command in the resturl log. Default: false
query	A request to the server to limit the items returned according to the query string. Example: If you are working with an ALM/QC data source, add the following row to extract only requirement folders: query=type-id[=1]
rawdata	Produces a separate log of each rest url response in json format. Default: false
resturls	Directs the extractor to produce a log of the rest queries run in sequence. Default: false
scopeddown	Specific to Rally. Directs the engine to return child project artifacts in addition to the selected project . Default: false
scopedup	Specific to Rally. Directs the engine to return parent project artifacts in addition to the selected project. Default: false
typemetadata	Queries and writes out the list of all types supported on a server. Default: false

Back to top

Fix

The Fix script cleans up any corrupt or stale data in the database that is the consequence of an upgrade from earlier versions of Micro Focus Connect.

This script typically is not used on a regular basis. It is meant for performing a cleanup after an upgrade.

Back to top

Maintenance

The Maintenance utility can be used to move project maps from one connection into a new connection, within the same application instance. Run the utility using the following syntax: java -jar mfcMaintenance.jar.

For a typical connection, use the following options to move project mappings from one connection to another. To see a list and description of all of the available options, run the utility with no options.

The following example looks for a synchronization named Connection A that has project mappings source1:target1 and source2:target2, and moves these mappings into a new synchronization named Connection B.

When using the Maintenance utility, follow these guidelines:

• To move multiple project maps, repeat the projectMapping option:

  -projectMapping "source1:target1" -projectMapping "source2:target2"

• If a project name contains a colon (such as in Jira), precede it with the backslash escape character, ‘\’. For example, to use the name source:1 add a backlash before the colon:

  -projectMapping "source\:1:target\:1" . This example also shows a non-escaped colon, as you only need the escape colon within the project name.

Back to top

MatchByName

Different products use different models for representing releases and sprints. For example:

• In Jira, sprints are independent of releases.
• In ALM Octane, sprints are children of releases.
• In QC/ALM, release cycles are children of releases, which are contained in release folders.

• In Azure DevOps, there is no explicit concept of a sprint. The closest are iterations, which are described as tree structures.

The different models make it impractical to synchronize release and sprint entities in one product with their equivalents in the other product.

Instead, we recommend that users manually create releases and sprints (or their equivalents) in the synchronized projects. The release and sprint names in both projects should be identical, including their letter cases.

The MatchByName scripts build tables of matching IDs for the identical releases and sprints in the synchronized projects.

With the tables of matching IDs, Micro Focus Connect can synchronize field values in items, such as stories and defects, that reference releases and sprints.

The following scripts are available for mapping data types between selected pairs of products:

• matchByNameAlmQcOctane

• matchByNameAzureDevOpsAlmQc

• matchByNameJiraAlmQc

• matchByNameJiraOctane

The following tables describe how the scripts map the data types between pairs of synchronized products:

When run, the script explores the Micro Focus Connect database as follows:

1. Discovers data sources based on the products specified in the script.
2. Discovers the connections that use the data sources.
3. Discovers the mapped source and target projects in the connections.
4. Between the source and target projects, creates tables of matching IDs for identical pairs of releases and sprints (or their equivalents).

You can choose to run the scripts according to the following schedules:

• Nightly, as part of the Purge script, so that the Micro Focus Connect database can catch up with any newly-added releases or sprints (or their equivalents).
• On demand by the Micro Focus Connect administrator, based on advice from stakeholders, when new releases or sprints are added to any of the synchronized projects.

Back to top

Connection scripts

The following scripts allow you to work with existing connections.

If the script contains a placeholder value, for example, ConnectionName, you must edit it in a text editor and replace the placeholder value with your own value.

Script name	Arguments	Description
RunOneIteration	-	Run connections in sequence—not simultaneously or in parallel. This capability helps you from exceeding your network bandwidth, and/or rate limiting or overload on the part of the source and target endpoint systems.
DeleteConnection	ConnectionName The name of the connection that you want to delete (case-sensitive), as displayed in Micro Focus Connect.	Deletes the specified connection. Replace the placeholder variable, ConnectionName. Caution: After a connection is deleted, all synchronization cross-references associated with this connection are also removed.
MergeConnections	ConnectionName1, ConnectionName2The names of the connections that you want to merge (case-sensitive), as displayed in Micro Focus Connect. ConnectionName3: A name for the merged connection.	Merges two connections into a single connection. You can merge two connections that have the identical master and target data sources. The script merges the types, projects, synchronization criteria, calculated values and cross references. The new connection is created in a disabled state. After your verify that the resultant connection was created correctly, delete the old connection and enable the new one.
StartAllConnections	-	Starts all connections.
mfcStopAllConnections	-	Stops all connections.
StartConnection	ConnectionName The name of the connection that you want to start (case-sensitive), as displayed in Micro Focus Connect.	Starts the specified connection.
StopConnection	ConnectionName The name of the connection that you want to stop (case-sensitive), as displayed in Micro Focus Connect.	Stops the specified connection.

Back to top

Purge scripts

The Purge script performs the following actions:

1. Deletes all iteration data from the Micro Focus Connect database.
2. Removes old logging and audit data from the database and log files.
3. Purges the dead space from the file system.

Recommended configuration

Set the script to run as a nightly, scheduled batch process, preferably at a time that causes the least inconvenience to users. If run every night, the script should require no more than 15 minutes of Micro Focus Connect server downtime. For details, see Purge files on Windows as a nightly task.

Stopping and restarting the Micro Focus Connect server clears out the process runtime memory cache.

Additional uses

You can use the Purge script as a general purpose utility to perform additional tasks that you need to run on a nightly basis. These include:

• Data folder backups

• Automatic clearing of watermarks across all connections

• Extracting users from endpoint systems and importing them into the Micro Focus Connect database user maps

• Stopping all connections, then starting and running them, one connection, one iteration at a time

• Stopping all connections, then starting each connection, one at a time

• Running custom scripts outside the context of Micro Focus Connect

Additional purge scripts

The following purge scripts are also available:

Script	Purpose
PurgeDisabledProjects	Removes all disabled projects across all connections from the Micro Focus Connect database. Note: To target a single connection, include the connectionname parameter.
PurgeDisabledTypes	Removes all disabled types across all connections from the Micro Focus Connect database. Note: To target a single connection, include the connectionname parameter.
PurgePreviousPasswords	Removes all previous passwords across all authenticator users from the Micro Focus Connect database.

Purge script effects on resources

When the Purge script runs, it extracts the audits stored in the Micro Focus Connect database since the last purge run. It also produces a .zip backup of the data folder, containing the most recent snapshot since the last purge. If SMTP email is configured, it also emails a .zip attachment of the audit extraction, to the addressees that were configured.

The size of the audit file and its content are directly affected by the success or failure of the Micro Focus Connect configuration. The more errors recorded in the audits, the larger the audit file. In general, we recommend increasing your disk space beyond the minimum required size since Micro Focus Connect is a resource intensive service. For details, see Windows installation.

The purpose of the audit file productions is to review and investigate errors and determine how to solve them. If the Micro Focus Connect synchronization errors are not relevant for your environment, you may turn off the audit file.

To turn off the auditing:

1. Open the mfcpurge.bat file, and rem out the line that calls audit.jar. Save the file.

2. Run the Purge script or add it to your automated tasks.

3. Set up a nightly job to delete all older audit and data .zip backups, for example seven or more days old. For details, see Purge files on Windows as a nightly task.

Back to top

Audit script

The Audit script generates a de-normalized flat .txt file representation of the audit UI log entries using the pipe character, |, as the column separator.

The file is given a timestamp and saved in the Connect/AppData/backup folder. The file can be imported into a spreadsheet or an SQL database for backup, querying, reporting, or general processing.

Row subset

You can produce a subset of the audit rows by specifying the connections by their name. Use the following syntax: connectionnames=connectionname1,connectionname2,… parameter

If you use the default and do not specify connection names, you receive the rows from all of the connections.

Column subset

You can produce a subset of the audit columns by specifying the column names. Use the following syntax: columnnames=columnname1,columnname2,… parameter.

If you use the default, and do not specify column names, you get all of the columns.

The following is an alphabetical list of the available columns : action, actionvalue, associatedid, endtime, fieldname, fieldvalue, itemactionid, itemid, iterationid, starttime, stacktrace, sourceproject, sourcetype, syncguid, targetproject, and targettype.

Time-based subset

You can produce a subset of the audit rows using the datetime=yyyy-mm-dd parameter. You can specify the data after a specific date. For example datetime=2022-01-03 returns rows that were added after January 3, 2022.

If you use the default, and do not specify a date and time, you get all of the data.

Syntax examples

The following examples show common usages of this utility.

• This example produces the default report, including all connections with all of the column names:

  "%JRE_PATH%\java.exe" -jar "%UTILITIES_PATH%\mfcAudit.jar" "%CONNECT_PATH%" "|" "%BACKUP_PATH%" “connectionnames=” “columnnames=” "datetime="

• This example produces a report for the connection named ‘CONN’ with a subset of the columns:

  "%JRE_PATH%\java.exe" -jar "%UTILITIES_PATH%\mfcAudit.jar" "%CONNECT_PATH%" "|" "%BACKUP_PATH%" “connectionnames=CONN” "columnnames=starttime,sourceproject,itemid,fieldname,fieldvalue,actionvalue" "datetime="

Back to top

QueryTemplates

The QueryTemplates script targets the running Micro Focus Connect server.

This script does not stop or restart the server, as it does not run direct queries against the Micro Focus Connect database.

This script is a template to query types, properties, and property lists from the data sources.

It is useful when discovering case sensitive property names and/or enum names to set up synchronization criteria and calculated values.

This script is useful for debugging issues with mappings, and is often used by support engineers when working one-on-one with the customer.

Back to top

ResetPassword

The ResetPassword script resets the password of the Micro Focus Connect Administrator user, replacing it with the string changeme.

This script is especially useful when a customer is locked out of Micro Focus Connect.

A subsequent login through the UI requires changeme to be the specified password, and Micro Focus Connect immediately asks the user to provide a new password.

Back to top