Improve object identification

This topic describes the features that enable you to improve object identification.

Identify objects by their text

You can use an object's name, or label, in its description to identify the object uniquely. This is the object's Identification text.

For example: Click the 'Buy Now' button. Using the Buy Now text in this step to describe the object, ensures that the correct button in clicked.

Sometimes, multiple text strings are identified near the object, and you can specify which one to use for object identification.

You can type the step in the editor or compose it in the step editing pane. The sections below explain the various options available when describing an object by its text.

Use rules and regular expressions

In some cases, you want to identify an object or text whose value varies, such as price, count, or ID. In this case, instead of specifying the exact value of the text, you can specify a rule or regular expression the text should match.

You can select one of the following rules:

contains
does not contain
begins with
ends with
Regular expression. Use this option if you need a more complex rule. Specify a regular expression pattern, using the Python regular expression syntax. Regular expressions are a powerful way of searching for and identifying sequences of characters by using specific patterns that the text being considered should match.

For syntax details and examples, see Identify an object by text.

To use a rule for the identification text:

In the step editing pane, click the Settings button near the object's identification text and select Use rule.
Select the type of rule from the list. The type can be: Contains, Does not contain, Begins with, Ends with, or Regular expression.
Enter the value for the rule or regular expression. To use a parameter, see Pass a value to the script.

To stop using a rule, click the x in the Use rule box.

Exact text matching versus AI matching

By default, AI algorithms are used for text matching. These may consider spelling variations and words with similar meaning to be matches.

If necessary, you can specify that the object's identification text must be matched exactly in order to identify the object.

In the step editing pane, click the Settings button near the object's identification text and select Exact match. To remove the Exact match setting, click its x.

Note:

The text in the application is identified using OCR. If the OCR is inaccurate, AI text matching may be more successful than exact text matching.
Exact text matching is case-insensitive.

Turn off Identification text

By default, objects are described using their Identification text.

To describe an object whose text varies in the application, you might want to use the object's type and position or relation, ignoring its text.

To do this, omit the text for identification from the step you type in the script, or turn off the Identification text toggle in the step editing pane.

This button is available when editing steps, when adding steps from inspection, and when describing a related object.

For more details, see Step syntax and Manually edit steps.

Text recognition in multiple languages

Optical Character Recognition (OCR) technology is used to recognize text in different languages in an application.

English is the default OCR language. However, you can add additional OCR languages to recognize text in other languages. For details, see Customize settings.

You can add the following OCR languages:

Afrikaans, Arabic, Basque, Bulgarian, Catalan, Simplified Chinese, Traditional Chinese, Croatian, Czech, Danish, Dutch, Esperanto, Estonian, Finnish, French, German, Greek, Hebrew, Hungarian, Icelandic, Italian, Irish, Japanese, Korean, Latin, Latvian, Lithuanian, Macedonian, Maltese, Norwegian, Polish, Portuguese, Romanian, Russian, Serbian, Slovenian, Slovak, Spanish, Swedish, Turkish, Ukrainian, Welsh.

Uniquely identify objects by location

To run a step on an object, the object must be identified uniquely. When multiple objects match your object description, you can information to make it unique.

Add one of the following:

A Position describes the object's ordinal location, and the direction in which to count. For example, Top 3.
A Relation let's you describe your object by its proximity to another object in the application.

Adding a relation may help create more stable tests that can withstand predictable changes to the application's user interface. Select related objects that are expected to maintain the same relative location to your object over time, even if the user interface design changes.

For example, you might want to add a step that removes an item from shopping cart that contains several items. Each item has its own Remove button.

You can describe the button to click as the Remove button that has the '<item name>' related object to its left.

For details on adding positions and relations, see Manually edit steps.

Use custom objects

To support an object type that Design does not recognize, you can create a custom object, which you describe using an image of the object.

You can use custom objects in script steps, as you would use built-in objects. You can also add a position or relation to custom object steps, to identify the object uniquely.

Note: Custom objects are not detected by Inspection and cannot be used as related objects.

To create a custom object:

In the Design Assets pane, click the Add new button in the Custom Objects node.

Tip: Collapsing the Scripts and Functions nodes may help bring the Custom Objects node into view.
Provide an image of the object that can be used to identify it.
- The image must be of type .png or .jpg, and its size must not exceed 1 MB.
- The image you use to describe the object must match the object exactly.
- Make sure that the image dimensions are similar to those displayed in the application.
Enter a class name for the custom object. Do not use a name of an existing object class.

The custom object is added in the Design Assets pane, and can be used for the object class in script steps.

To view the list of custom objects:

In the Design Assets pane, expand the Custom Objects node.
In a script's step editing pane, open the Object class list.

To view custom object details:

Open the Design Home page.
In the Design Assets pane, select the Custom Objects node.
View custom object details, such as an object's image, class name, and image dimensions.

To edit or delete custom objects:

Open the Design Home page.
In the Design Assets pane, select the Custom Objects node.
Select one or more custom objects and click Edit or Delete. You can edit the class name, and remove or replace the image used to identify the object.

Modifying or deleting a custom object affects scripts that use this object. The scripts are not updated automatically. Review and update your scripts manually to accommodate the modifications.

Automatic scrolling

When a script runs, if an AI object is not displayed in the visible part of the web page, the script auto-scrolls further in search of the object. After the object is identified, no further scrolling is performed. This prevents failures during a run session.

By default, the script scrolls down twice. You can customize the direction of the scroll and the maximum number of scrolls to perform, or deactivate scrolling if necessary. For details, see Customize settings.