AI-based testing in UFT One

Mobile, web, and windows-based SAP GUI tests

This topic explains how to use UFT One's Artificial Intelligence (AI) Features in your tests to identify objects the way a person would. This enables you to run the same test on different platforms and versions, regardless of the objects' implementation.

Note: AI Codeless Testing is not part of UFT One. It is a standalone tool that provides AI-based functional testing for UFT Model-based Testing. For details, see the AI Codeless Testing Help Center.

AI-based testing overview

UFT One's AI Features enable your tests to interact with the application you are testing in the same way a person would. UFT One uses AI to identify objects visually, based on a wide variety of images, context, and sometimes text.

For example, UFT One AI can identify many types of search fields, user profile areas, input fields, buttons, shopping carts and more.

Some advantages of AI-based object identification are: 

Easier to edit

Test scripts are more intuitive.
Test multiple environments

Tests are technology agnostic, identifying objects visually, regardless of the technology details used behind the scenes.
Test resilience

Tests are easier to maintain, as an object changing location, framework, or even shape, won’t break the test script as long as the object remains visually similar or its purpose remains clear.

AI service

The AI services that provide UFT One's AI Features are computationally intensive. We recommend that you use a powerful computer to benefit from the AI Features and achieve optimal performance. For recommended system requirements, see Support Matrix.

You can install the remote AI service on one powerful computer, letting it provide AI capabilities for multiple weaker UFT One computers. For details, see Use the remote AI service.

Back to top

Prerequisites

To use UFT One's AI Features, they must be installed and enabled. By default, UFT One's installation is set up to install and enable UFT One's AI Features.

If the AI Features are not installed (the Options dialog box does not display the GUI Testing > AI pane), do one of the following: 

To enable or disable the AI Features after installation, use the Tools > Options > GUI Testing > AI > AI active option.

Prepare your test solution and create a test for a web, mobile, or SAP GUI for Windows application.

To start identifying AI objects to use in your test, continue with one of the following tasks:

Back to top

Inspect your application for objects

Use AI inspection to identify objects in your application that you can use in your test steps.

Prerequisites

If you are testing an application on a mobile device

  1. Make sure the Mobile Add-in is installed and loaded.

    To test web applications on your mobile device, make sure the Web Add-in is also installed and loaded.

  2. Connect to UFT Mobile and open your mobile device and the application you want to test. For details, see Open a remote access window and application manually.

If you are testing a desktop web application

  1. Make sure the Web Add-in is installed and loaded.

  2. Make sure that your browser is set up with the UFT extension. For details, see Test Web apps on specific browsers.

  3. Open the application.

    For best identification results:

    • Maximize the browser window and set the zoom level to 100%.

    • Set your Windows display settings to display all text and apps at 100%.

    • Make sure no banners are displayed in the browser aside from your web page.

Supported browsers: Chrome, Chromium-based Edge, Firefox, Internet Explorer.

Headless browsers are not supported.

If you are testing an SAP GUI for Windows application

(UFT One version 2022 and later)

  1. Make sure that you perform the following tasks as described in Enable support for SAP GUI for Windows.

    • Install SAP GUI Scripting.

    • Enable SAP GUI Scripting on the server side.

    • Enable SAP GUI Scripting on the client side.

  2. Make sure the SAP Solutions Add-in is installed and loaded.

  3. Open the application.

Note: AI-based testing for SAP GUI for Windows applications is officially supported on the Classic and Signature themes.

To identify single objects

In the Object Spy or Object Identification Center, as you spy on objects in mobile, web, or SAP GUI for Windows applications, UFT One also presents the AI object you could use to represent the same object.

When spying on a single object Once you click on an object in the application, an AI object is suggested only for the object you clicked. AI objects are not suggested for any other object in the hierarchy displayed in the Spy.

When spying on multiple objects using the OIC

AI object suggestions are available in the Edit view of each spied object. To open the edit view, hover over an object in the list of spied objects and click the Edit button.

To identify all objects

  1. Open the AI Inspection window to inspect your application and detect all AI objects in it.

    To open the window, do one of the following: 

    • Select AI > AI Identification.

    • Click the AI Identification toolbar button.

      Tip: Depending on the version you are using, the button might look different.

    • In the Object Spy or Object Identification Center, click INSPECT.

      Tip: This button is available when the AI Auto Inspection option is enabled.

  2. Click a web application, an SAP GUI for Windows application, or the remote access window displaying your mobile application. If you click an unsupported context or press Esc, the Select Application button appears to prompt you to click again.

    Note: You can skip this step if you began the inspection with a web Browser, mobile Device, or SAPGUISession test object as the top-level object in your spied hierarchy. UFT One automatically selects that test object for inspection.

    The Live Application tab displays the current screen of the application, highlighting all of the detected objects.

  3. Decide whether to show Visual Elements, Texts, or both, to see either the objects that UFT One detected visually, or areas of text in the application, or both.

  4. (Tech preview) If your application contains container objects such as tables and you want to inspect the lower-level objects inside the tables, toggle on Containers to initiate a reinspection. (The switch name may vary between versions.)

    Note: Table cell objects are identified using ABBYY OCR.

    UFT One versions 2021 R1 and later: To identify TableCell objects, you must have the ABBYY OCR Engine feature installed in UFT One.

  5. After identifying the AI objects, you can Add AI-based steps to your tests.

Help design the future of AI-based testing in UFT One

Click How is the detection? Help us to improve to open the Feedback Tool and send Micro Focus feedback about the object detection. For details, see AI-based testing Feedback Tool.

Back to top

Inspect application mockups for objects

Use AI Mockup Identification to inspect application mockups and identify objects to use in your test. This enables you to design and prepare your test even before your application is fully developed.

Prerequisites

AI Mockup Identification does not require a ready application. Instead, create a local folder containing images in .jpg, .jpeg, or .png format.

Inspect application mockups

  1. In the UFT One toolbar, click the down arrow near AI Identification button , and select AI Mockup Identification.

  2. In the MOCKUP IMAGES tab of the AI Inspection window, select the inspection context and click Browse Folder to select a folder containing images.

    Inspection contexts: Non-Mobile/Mobile or Web/Mobile depending on your UFT One version.

    The AI Inspection window inspects the image that appears first in the file name order and highlights all identified visual elements by default.

  3. Decide whether to show Visual Elements, Texts, or both, to see either the objects that UFT One detected visually, or areas of text in the application, or both.

  4. (Tech preview) If your application contains container objects such as tables and you want to inspect the lower-level objects inside the tables, toggle on Containers to initiate a reinspection. (The switch name may vary between versions.)

    Note: Table cell objects are identified using ABBYY OCR. UFT One versions 2021 R1 and later: To identify TableCell objects, you must have the ABBYY OCR Engine feature installed in UFT One.

  5. Decide whether to change the AI identification context type by selecting Non-Mobile or Mobile.

  6. Click the down arrow near the folder icon to select a different folder, synchronize with the current folder, or go to the current folder.

  7. To view all images in your folder, click the down arrow on the top to show the image gallery. In the gallery, you can:

    • Navigate between images in the folder with arrows.

    • Click the Search button to search for a specific image.

    • Select whether to view the images in Grid View or Line View.

    • Select Name or Date to sort images by the file name or modified time.

  8. After identifying the AI objects, you can Add AI-based steps to your tests.

Help design the future of AI-based testing in UFT One

Click How is the detection? Help us to improve to open the Feedback Tool and send Micro Focus feedback about the object identification. For details, see AI-based testing Feedback Tool.

Back to top

Add AI-based steps to your tests

To add AI-based steps in your tests:

Set the AI context

Before you can run any AI-based steps in your test, you must set the context in which to use AI.

The AI context is a UFT One test object of one of the following:

UFT One versions 2022 and later Web Browser, mobile Device, or SAPGUISession.
UFT One versions 2021 R1 and earlier: Web Browser or mobile Device.

Within this context, UFT One searches for objects using AI. You can set the AI context for your test in one of the following ways:

  • In the Live Application tab of AI Inspection window, add a SetContext step from the right pane, or click the down arrow near Context and select Add as "SetContext" step to test.

  • Manually add a SetContext step to your test before you add any other steps. This is necessary when you create a test script for application mockups.

  • A SetContext step is automatically added if you use AI Record to record a test.

UFT One must be able to identify the context object throughout the test run. If necessary, use additional SetContext steps to select a different object for the context.

Tip: Within a web browser context, you can also add AIUtil.Context.SetBrowserScope steps, to specify whether to inspect the whole browser for objects or only the web page itself. Inspecting the whole browser is useful when your test needs to respond to a pop-up dialog box that opens in the browser. Note that UFT One cannot identify any pop-up dialog boxes hidden behind the browser window.

Add test steps

You can add AI-based steps to your test in one of the following ways:

  1. Make sure that you have inspected a page or screen of your open application in the AI Inspection window. See To identify all objects.

  2. Click a highlighted object.

    A pop-up dialog box opens with the suggested step and object description.

  3. In the pop-up dialog box, modify the step action, edit the value field, and click Add to test.

    Alternatively, click Edit step to add additional object description properties including Position and Relation in the right pane.

    UFT One version 2022 and later

    • You can specify properties that do not each uniquely describe the object. You can only add the step to your test if the combined description uniquely identifies the object. Otherwise, when you move to another object, your edits are discarded, and the original description is used.

    • When you add a relation to an object, you can see and edit the relation direction used in your object description.
    UFT One version 2021 R1 and earlier

    Each property you include in the description must uniquely describe the object. Otherwise, you cannot edit other properties, and you cannot save the description.

    If the object can only be described uniquely by multiple properties, create the description in the test script.

    • For an object with more than one identified text option, UFT One performs verification on the text value you select.

    • For cells inside a table, the pop-up dialog box also displays the object hierarchy, row, and column information.

    • Selecting items from a combo box is a complex operation. The Select action is supported on a combo box identified by AI only if UFT One can also identify the combo box as a WebList or a MobileDropDown.

    • You can describe an object by position or relation, but not both.

  4. After you add a step to your test, click the Run on application button in the right pane to run the step. This verifies that your step is correct, and also advances the application to the next state, re-inspecting the application for the next step.

    You can instruct UFT One to provide time for the application to load after running the step, before re-inspecting the application. Click the dot-menu icon next to the Run on application button and configure a delay.

  5. To inspect the whole browser window, enabling the test to handle browser pop-ups or prompts, do the following:

    1. Click the down arrow near Context , and select Set browser scope to full window .

    2. Click the browser window and then go back to the AI Inspection window.

      The AI Inspection window starts inspecting the whole browser window.

    3. Add a SetBrowserScope step from the Context menu or right pane to set the context.

The step added for each object includes any information used to identify it uniquely, such as associated text or the object's relative location on the screen. For details, see Associate text with objects and Identify objects by relative location.

Note: Avoid adding steps with objects that were incorrectly identified. For example, if a button is identified as a text box, or a check mark is identified as a button, such objects may be identified inconsistently and fail in subsequent test runs.

  1. Make sure that you have inspected an application mockup in the AI Inspection window. See Inspect application mockups for objects.

  2. Click any highlighted object.

    A pop-up dialog box opens with the suggested step and object description.

  3. In the pop-up dialog box, modify the step action, edit the value field, and click Add to test.

    Alternatively, click Edit step to add additional object description properties including Position and Relation in the right pane.

    UFT One version 2022 and later

    • You can specify properties that do not each uniquely describe the object. You can only add the step to your test if the combined description uniquely identifies the object. Otherwise, when you move to another object, your edits are discarded, and the original description is used.

    • When you add a relation to an object, you can see and edit the relation direction used in your object description.
    UFT One version 2021 R1 and earlier

    Each property you include in the description must uniquely describe the object. Otherwise, you cannot edit other properties, and you cannot save the description.

    If the object can only be described uniquely by multiple properties, create the description in the test script.

    • For an object with more than one identified text options, UFT One performs verification on the text value you select.

    • For cells inside a table, the pop-up dialog box also displays the object hierarchy, row, and column information.

    • You can describe an object by position or relation, but not both.

Drag and Drop the identified object from the Object Spy or Object Identification Center into your test.

An AI-based test step is automatically created.

Use AI Record to automatically create AI-based test steps in your test editor.

For details, see Record an AI-based test.

Edit your test using the AIUtil object.

Next, use the AIUtil object to identify the control by type, text, or both, and to perform operations on the objects, such as Click, Type, or Search.

For details, see AI-based Testing in the UFT One Object Model Reference for GUI Testing.

Example:  

The following example clicks a settings control on a mobile device and then types a user name in the appropriate field. Both controls are identified visually using AI.

Next, the Log In text is located on the screen and clicked.

After logging in, this example clicks an "ON" button. Since the application contains a few similar buttons, the example specifies that button to click is the third button from the top.

AIUtil.SetContext Device("device") 'Set the context for AI
AIUtil("settings").Click
AIUtil("input", "USER NAME").Type "admin"
AIUtil.FindTextBlock("Log In").Click
AIUtil("button", "ON", "FromTop", 3).Click

Tip: UFT One generates steps that use FindTextBlock, as the spy identifies full blocks of text. When you create your own steps, consider using the FindText method, which will match the text you provide, even if it is only part of a larger text block.

Add checkpoints to your test (optional)

You can add different types of checkpoints to check the existence of an object or its state.

Add checkpoint steps in one of the following ways:

  • Enter checkpoint steps in your test manually.

  • Add checkpoint steps in an AI Record session. For details, see Add a checkpoint during recording.

  • Select Verify as the action and select Exists or Does Not Exist to verify the existence of the object. Use On or Off to verify the object's state when you edit a step in AI Inspection window.

Checkpoint method Description Condition

AIObject.CheckExists or AITextObject.CheckExists

Check the existence or absence of an object in your application.

 Specify whether you expect the object to exist or not when this step runs.
AIObject.CheckState

Check the state of objects that can be set to Off or On, such as check boxes, toggles, or radio buttons.

 Specify the expected state of the object.

The checkpoint passes if the application is in the expected conditions. Otherwise, a step failure is reported in the run results.

Back to top

Inspect the next application page/screen

When you finish creating test steps for one page or screen in your application and you want to continue on another one, follow the steps below:

To inspect the next application page/screen

  1. Click the down arrow near Context , select Go to inspected application , and navigate to the desired location in the application.

  2. In the AI Inspection window, click Re-inspect to load the new application page or screen and reinspect it.

    • If you don't reload and try to run a step from the AI Inspection window, UFT One runs the step based on the previous page’s inspection, resulting in an error or performing the operation on the new page.

    • If multiple remote access windows or browser windows are open, the inspection session interacts with only one.

    • You can switch to inspect other browser windows or mobile devices that are open on your computer. Click the down arrow near Context and then select Select a different application in the AI Inspection window.

    • If you need to perform steps on the application to prepare it for inspection, you can use the Delayed re-inspect.

      Click the down arrow near Delayed re-inspect and set the required delay.

      Click Go to begin the countdown, open the application, and perform steps such as hover or menu clicks to bring the application to the state you want to inspect.

      When the delay timer expires, the application is re-inspected.

  3. Add steps from the new page or screen to your test.

Tip: You may need to set the delay to a larger value if you have a slow network.

Back to top

Run AI-based tests

After inspecting your application and creating test steps, run your AI-based test as you would run any other UFT One test. See Run / Debug tests.

You can run the same test on different operating systems and versions since it is not based on implementation details.

Back to top

Convert technology-based tests to AI-based tests

The AI Transformation Assistant can help you convert property-based test steps in your existing tests to AI-based testing steps:

  1. Enable the assistant.

    In Tools > Options > GUI Testing > AI > General, select Replace run results with AI object suggestions.

  2. Run your tests.

    The test steps are performed on the application and the run report contains AI object suggestions instead of the results of your test run. Each step contains a suggestion for transforming the technology-based test step into an AI-based testing step.

    Note: The AI Transformation Assistant does not check the object hierarchy inside table or calendar objects, and does not suggest AITable or AICalendar containers.

  3. Modify your test steps (AI conversion).

    Replace existing steps with the suggested AI object steps to create a more resilient and platform-agnostic test.

Back to top

Troubleshooting

AI text identification requires the Windows mediaserver.exe service to be running. Otherwise the following may occur:

  • AI inspection cannot find objects By Text.
  • All FindText and FindTextBlock steps fail to return an AITextObject when running a test.
  • An error message indicates that an error occurred when calling the Media Server OCR service.

Solution: 

Open the Windows Services Manager and make sure the mediaserver.exe is running. Otherwise, start the service manually.

Back to top

See also: