Improve AI-based test object identification

GUI UFT Mobile and Web tests only

This topic discusses some of the elements that UFT One's Artificial Intelligence (AI) Features use to support unique identification of objects in your application.

Associating text with objects

The text associated with an object can help identify an object uniquely. For example:

AIUtil("input", "USER NAME").Type "admin"

Using the USER NAME text in this step to describe the object, ensures that we type in the correct field.

When detecting objects in an application, if there are multiple labels around the field, UFT One uses the one that seems most logical for the object's identification.

However, if you decide to use a different label in your object description, UFT One still identifies the object.

Example: When detecting objects in an application, a button is associated with the text on the button, but a field is associated with its label, as opposed to its content.

If a field has multiple labels and UFT One chooses one for detection, UFT One will still identify this field correctly when running a test step that uses a different label to describing the field.

In some cases, UFT AI combines multiple text strings that are close to each other into one text string to identify one object.

You can edit the combined string and leave just one to use for object identification. Make sure to remove a whole string and not part of it.

Example:  

For the text box below, UFT AI combines Password and Forgot? into one string to identify the object and adds the following step code to your test script:

AIUtil("text_box", "Password Forgot?").Type

You can remove a whole string from your script and change the code to the following without causing test failure:

AIUtil("text_box", "Password").Type

Back to top

Configure text recognition options

Customize these text recognition options to reach optimal text recognition on your application:

Text recognition in multiple languages

To enable UFT One's AI features to recognize text in languages other than English, select the relevant OCR languages in Tools > Options > GUI Testing > AI. See AI Pane (Options Dialog Box > GUI Testing Tab).

Exact text matching when identifying objects using AI

AI-based object identification uses AI algorithms for text matching. AI text matching may consider spelling variations and words with similar meaning to be matches.

If necessary, you can temporarily instruct UFT One to look exactly for the specified text when identifying AI test objects.

Add an AI run settings step to your test to set the text matching method. The setting is effective until you change it or until the end of the test run. For details, see the AIRunSettings Object in the UFT One Object Model Reference for GUI Testing.

Note:  

  • The text in the application is identified using OCR. If the OCR is inaccurate, AI text matching may be more successful than exact text matching.
  • Exact text matching is case-insensitive.

Use the UFT One text recognition settings

Instruct AI to use the UFT One text recognition settings. Depending on your UFT One configuration, this enables you to use Google or Baidu OCR cloud services or other UFT One OCR engines. See AI Pane (Options Dialog Box > GUI Testing Tab).

Back to top

Identifying objects by relative location

UFT One must be able to identify an object uniquely to run a step on the object. When multiple objects match your object description, you can add the object location to provide a unique identification. The location can be ordinal, relative to similar objects in the application, or proximal, relative to a different AI object, considered an anchor.

The AIUtil object identification methods now use location and locationData arguments in which you can provide information about the object's location. See AIObject, FindText, and FindTextBlock.

To describe an object's ordinal location

Provide the following information:

  • The object's occurrence, is it first, second, third, and so on.

  • The orientation, the direction in which to count occurrences: FromLeft, FromRight, FromTop, FromBottom.

For example, AIUtil("button", "ON", "FromLeft", 2).Click clicks the second ON button from the left in your application.

To describe an object's location in proximity to a different AI object

Provide the following information:

  • The description of the anchor object.

    The anchor must be an AI object that belongs to the same context as the object you are describing.

    The anchor can also be described by its location.

  • The direction of the anchor object compared to the object you are describing: WithAnchorOnLeft, WithAnchorOnRight, WithAnchorAbove, WithAnchorBelow.

UFT One returns the AI object that matches the description and is closest and most aligned with the anchor, in the specified direction.

For example, this test clicks on the download button to the right of the Linux text, below the Latest Edition text block:

Set latestEdition = AIUtil.FindTextBlock("Latest Edition")
Set linuxUnderLatestEdition = AIUtil.FindText ("Linux", micWithAnchorAbove, latestEdition)
AIUtil("button", "download", micWithAnchorOnLeft, linuxUnderLatestEdition).Click

Back to top

Describe a control using an image

If your application includes a control type that is not supported by AI identification, you can provide an image of the control that UFT One can use to identify the control. Specify the class name that you want UFT One to use for this control type by registering it as a custom class.

AIUtil.RegisterCustomClass "door", "door.PNG"
AIUtil("door").Click 

For more details, see the AIUtil.RegisterCustomClass method in the UFT One Object Model Reference for GUI Testing.

Once you register a custom class, you can use it in AIUtil steps as a control type:

  • AIUtil("<Myclass>")
  • AIUtil("<Myclass>", micAnyText, micFromTop, 1)
  • AIUtil("<Myclass>", micAnyText, micWithAnchorOnLeft, AIUtil("Profile"))

You can even use it to describe an anchor object, which is then used to identify other objects in its proximity:

AIUtil("profile", micAnyText, micWithAnchorOnLeft, AIUtil("<Myclass>") )

Note:  

  • You cannot use a registered class as an anchor to identify another registered class by proximity.
  • The image you use to describe the control must match the control exactly.

Back to top

Verify object identification

To increase the test run success rate, UFT One verifies the object identification before performing the operation. This is useful when objects in the application change locations during run time, for example, they are covered up by ads or scrolled out of the view.

By default, object identification verification is enabled for non-mobile contexts and disabled for mobile contexts. You can change the context settings in Tools > Options > GUI Testing > AI.

To enable or disable object identification verification for a specific test run, use the AIRunSettings.VerifyIdentification object. See the AIRunSettings Object in the UFT One Object Model Reference for GUI Testing.

Even if object identification verification fails during the test run, UFT One still performs the operation, and a warning message is displayed in the run result report.

Back to top

Automatic scrolling

When running a test, if the object is not displayed in the application but the web page or mobile app is scrollable, UFT One automatically scrolls further in search of the object. Once an object matching the description is identified, no further scrolling is performed. Identical objects displayed in subsequent application pages or screens will not be found.

UFT One scrolls similarly when running a checkpoint that requires that an object not be displayed in the application. The checkpoint passes only if the object is not found even when scrolling to the next pages or screens.

By default, UFT One scrolls down twice. You can customize the direction of the scroll and the maximum number of scrolls to perform, or disable scrolling if necessary.

  • Globally customize the scrolling in Tools > Options > GUI Testing > AI, or using a UFT One automation script. See AI Pane (Options Dialog Box > GUI Testing Tab).
  • Customize the scrolling temporarily within a test run using the AIRunSettings.AutoScroll object. See the AIRunSettings Object in the UFT One Object Model Reference for GUI Testing.

Note: You can also add steps within your test to actively scroll within the application. For details, see AIUtil.Scroll and AIUtil.ScrollOnObject in the UFT One Object Model Reference for GUI Testing.

Back to top

Update AI models

UFT One's AI engine uses AI models to facilitate AI-based testing activities. AI models are installed and activated automatically when you install and enable UFT One's AI features. However, you can update your models to new ones to improve AI performance and accuracy.

Note: If necessary, you can revert to earlier models you used before.

Micro Focus periodically updates the AI models. All models are available online on the Marketplace. When a new model is released, a notification pops up in the lower right corner of your desktop when you launch UFT One.

To update an AI model from UFT One user interface

  1. Prerequisite: In Tools > Options > GUI Testing > AI > Model, configure the Marketplace connection information.

    See Configure Marketplace connection information for AI model update.

  2. Select AI > AI Model Update in the toolbar.

    In the AI Model Update dialog box that opens, you can see the models you are using and available models you can download. The active models for both web and mobile contexts are always displayed at the top of each model table.

  3. Select an AI model and click Download.

    The Status column of the model shows the download progress.

    The download may take some time. You can close the dialog box and continue with your work. When the download is completed, a notification pops up in the lower right corner of your desktop.

  4. Click Activate model.

    After the new model is activated, it is used immediately by UFT One's AI engine. It will also be displayed at the top of the model table when you open the AI Model Update dialog box again.

To update an AI model from a command-line interface

To update the active AI model from a command line, run the ModelInstallApp.exe command. This is designed for users who have no network connection in UFT machines. You can download the model in your local machine and update for all UFT machines.

Preparation

Syntax

Run the following syntax in the command line tool:

ModelInstallApp.exe <Model file directory>

Model file directory: The path of the model zip file, for example, <The folder that stores the model file>\models.zip.

Back to top

See also: