AI-based testing in UFT One

GUI UFT Mobile and Web tests only

This topic describes how to install and use UFT One's Artificial Intelligence (AI) Features to identify objects the way a person would. This enables you to run the same test on different platforms and versions, regardless of the objects' implementation.

In this topic:

UFT One AI-based testing support

The following table describes the level of AI-based testing support provided in each UFT One version.

Back to top

AI-based testing overview

UFT One's AI Features enable your tests to interact with the application you are testing in the same way a person would. UFT One uses AI to identify objects visually, based on a wide variety of images, context, and sometimes text.

For example, UFT One AI can identify many forms of search fields, user profile areas, input fields, button, shopping carts and more.

Some advantages of AI-based object identification are: 

1. More intuitive test scripts.

2. Tests are technology agnostic, identifying objects visually, regardless of the UI technology details used behind the scenes.

3. Tests are easier to maintain, as an object changing location, UI framework, or even shape, won’t break the test script as long as the object remains visually similar or its purpose remains clear.

Back to top

Install UFT One's AI Features

To use UFT One's AI features, you must install and enable them.

Prerequisites

Before installing the AI features, make sure you have the following installed: 

• A 64-bit operating system

• Microsoft Visual C++ 2013 Redistributable

• The required Windows feature, depending on your Windows operating system:

  Windows OS - Microsoft Media Feature Pack

  1. From the Control Panel, select Programs and Features > Turn Windows features on or off.

  2. In the Windows Feature window, turn on Media Features.

  3. Restart the computer to take effect.

  Windows Server 2008 - Desktop Experience

  1. From the Control Panel, select Programs and Features > Turn Windows features on or off.

  2. In the Server Manager window, in the right pane, click Add Features.

  3. In the Add Features Wizard window, turn on Desktop Experience.

  4. Click Install.

  5. Restart the computer for your changes to take effect.

  Windows Server 2012 and later - Microsoft Media Foundation

  1. From the Control Panel, select Programs and Features > Turn Windows features on or off.

  2. In the Add Roles and Features Wizard window, click Next until you reach the Features tab.

  3. Turn on Media Foundation.

  4. Click Install.

  5. Restart the computer for your changes to take effect.

  For the most up-to-date instructions on enabling Windows features, see the Microsoft documentation.

Install UFT One's AI features

Expand one of the sections below, depending on the UFT One version you are using: 

UFT One 15.0.2 or later

When you run the UFT One installation program, AI Features is selected by default in the list of features to include in the installation. If you deselected it when you first installed UFT One, run the installation program again to Modify the installation and select AI Features to install.

Alternatively, run the UFT One silent installation and specify AI Features for installation using the ADDLOCAL parameter. For details, see Silent commands for installing individual UFT One features.

Note: If UFT AI Features 15.0.1 or earlier was previously installed, this installation removes the older one.

Once installed, AI Features are enabled by default. If you ever need to disable them, open Tools  > Options  > GUI Testing > AI, and clear the AI active option.

UFT One 15.0.1 or earlier

After installing UFT One, run the separate UFT AI Features installation program.

Use the version of AI Features that is delivered with your UFT One version.

• For UFT One 15.0, install UFT One AI Features version 2.0

• For UFT One 15.0.1, install UFT One AI Features version 15.0.1

You can run the installation manually, using the setup wizard or you can run it silently from a command line.

Manual installation	Silent installation
From the UFT One DVD, run AI\setup.exe installation file and follow the on-screen instructions. When selecting the features to install, you only need the AI Engine.	msiexec /qn /i "<UFT DVD>\AI\AI_Installer.msi" ADDLOCAL="AI_Services"

Enable UFT One's AI features

1. Open the Options dialog box to the AI pane of the GUI Testing tab (Tools > Options > GUI Testing > AI).
2. Select the AI active option.
3. Restart UFT One.

If you use the Generate Script option in the Options dialog box, the AI Active option is included in the script. For details, see General Pane (Options Dialog Box > GUI Testing Tab).

Add a statement to the generated script to restart UFT One if this option's value is modified. For details, see the Options Object > AI Property section of the UFT One Automation Object Model Reference.

Back to top

Inspect your application for objects

Use AI inspection to identify objects in your application that you can use in your test steps.

Prerequisites

Inspect your application

UFT One 15.0: Identify all objects on the device screen

Access the AI Spy from within the Object Spy:

1. In UFT One, click the Object Spy toolbar button to open the Object Spy.

2. In the Object Spy, click the AI spy .

3. Click on your mobile application in the remote access window.

   The spy's AI Identification View displays an image of the application, highlighting all of the detected objects.

4. You can switch between By Object and By Text modes, to see either the objects that UFT One detected visually, or areas of text in the application.

UFT One 15.0.1 or later: Identify single objects

In the Object Spy or Object Identification Center, as you spy on objects in mobile or web applications, UFT One also presents the AI object you could use to represent the same object.

When spying on a single object	Once you click on an object in the application, an AI object is suggested only for the object you clicked. AI objects are not suggested for any other objects in the hierarchy displayed in the Spy.
When spying on multiple objects using the OIC (15.0.2 or later)	AI object suggestions are available in the edit view of each spied object. To open the edit view, hover over an object in the list of spied objects and click the Edit button.

Drag the AI object into your test to create a step that performs an operation on that object.

To toggle off the AI suggestions, toggle off AI Auto Inspection in the Object Spy window or in the Object Identification Center settings.

Tip: This setting remains in effect in subsequent Spy sessions and subsequent UFT One sessions, until you toggle it on again.

UFT One 15.0.1 or later: Identify all objects on the device screen or web page

1. Open the AI Identification View window to inspect your application and detect all AI objects in it.

   To open the window, do one of the following: 

   • Click the AI Identification toolbar button.

     Tip: Depending on the version you are using, the button might look different.

   • In the Object Spy or Object Identification Center, click INSPECT.

     Tip: This button is available when the AI Auto Inspection option is enabled.

2. Click on a web application or on the remote access window displaying your mobile application.

   Note: You can skip this step if you began the inspection with a Web browser or Mobile device as the top-level object in your spied hierarchy. UFT One automatically selects that browser or device for inspection.

   The AI Identification View displays an image of the application, highlighting all of the detected objects.

3. You can decide whether to show Visual Element or Text, to see either the objects that UFT One detected visually, or areas of text in the application.

Help design the future of AI-based testing in UFT One

Click How is the detection? Help us improve to open the Feedback Tool and send Micro Focus feedback about the object detection. For details, see AI-based testing Feedback Tool.

Back to top

Inspect application mockups for objects

Supported on UFT One versions 15.0.2 and later

Use AI Mockup Identification to inspect application mockups and identify objects you could use in your test. In this way, you can prepare your test much earlier and design your test even before your application is fully developed.

Prerequisites

AI Mockup Identification requires no web or mobile application. Instead, you need to load images in .jpg, .jpeg, or .png format from a local folder.

Inspect application mockups

1. In the UFT One toolbar, click the down arrow near the AI Identification button and select the AI Mockup Identification .

2. In the dialog box that opens, select a folder containing images, select Web or Mobile as the AI identification context, and click Open.

   The AI Mockup Identification View loads all of the images in the folder by file name and creates an image gallery.

   The AI Mockup Identification View inspects the image that appears first in the file name order and highlights all identified visual elements by default.

3. To manage images in the folder, edit settings in the image gallery:

   • Select File Name or Modified Time to sort the images by file name or modified time.
   • Click Refresh to refresh the images if the images are changed.
   • Click Choose different to select a different folder of images to inspect.

4. To achieve optimal identification, edit settings on the inspected page:

   • Select Visual Element or Texts to show either the objects that UFT One identified visually or areas of text.
   • Select Web or Mobile to change your AI identification context again.

Help design the future of AI-based testing in UFT One

Click How is the detection? Help us improve to open the Feedback Tool and send Micro Focus feedback about the object identification. For details, see AI-based testing Feedback Tool.

Back to top

(Optional) Verify support for selecting from a combo box

Supported on UFT One versions 15.0.1 and later

Selecting items from a combo box is a complex operation. UFT One can perform a Select operation on a combo box identified by AI only if it can also identify this combo box as a WebList or a MobileDropDown.

When inspecting your application for objects using AI, if UFT One identifies a combo box, it displays a Verify icon on the object. Click this icon to check whether UFT One can run Select operations on this object before adding such steps to your test.

UFT One relies on the relative location of a combo box to verify the object. If you re-size the browser after inspecting a web page and change the object location, the verification fails.

Back to top

Add AI-based steps to your tests

Set the AI context

Before you can run any AI-based steps in your test, you must set the context in which to use AI. Add a statement to your test similar to this one: 

UFT One 15.0: The context must be a mobile Device test object that exists in your object repository.

UFT One 15.0.1 and later: The AI context can be one of the following: 

• A mobile Device test object stored in your object repository.

• A web Browser test object stored in your object repository or described using descriptive programming.

  The web Browser can run on a Desktop or Mobile device, and be opened manually or from the Record and Run settings.

Add test steps

You can add AI-based steps to your test in one of the following ways:

• Drag and Drop the identified Object or Text from the Object Spy, Object Identification Center, AI Identification View or AI Mockup Identification View window into your test.

  The step added for each object includes any information used to identify it uniquely, such as associated text or the objects relative location on the screen. See Associating text with objects and Identifying objects by relative location.

  Note: Avoid adding steps with objects that were incorrectly identified. For example, if a button is identified as a text box, or a check mark is identified as a button, such objects may be identified inconsistently and fail in subsequent test runs.

• Use AI Record to automatically create AI-based test steps in your test editor. For details, see Record an AI-based test.

• Edit your test using the AIUtil object.

  Next, use the AIUtil object to identify the control by type, text, or both, and to perform operations on the objects, such as Click, Type, or Search.

  For details, see AI-based Testing in the UFT One Object Model Reference.

AIUtil.SetContext Device("device") 'Set the context for AI
AIUtil("settings").Click
AIUtil("input", "USER NAME").Type "admin"
AIUtil.FindTextBlock("Log In").Click
AIUtil("button", "ON", "FromTop", 3).Click

Add checkpoints to your test (optional)

Supported on UFT One versions 15.0.2 and later

You can add different types of checkpoints, checking the existence of an object, or its state.

Enter checkpoint steps in your test manually, or add them in an AI Record session. For details, see Add a checkpoint during recording.

The checkpoint passes if the application meets the expectation. Otherwise, a step failure is reported in the run results.

Back to top

Inspect the next application page/screen

When you finish creating test steps for one page or screen in your application and you want to continue on another one, navigate to the desired location in the application itself. Then, in the AI Identification View window, click REFRESH to load the new application screen and re-inspect it.

• When working with a mobile application, navigate to the correct location in your application directly on the device or using the remote access window.

• If multiple remote access windows or browser windows are open, the inspection session interacts with only one.

  UFT One 15.0.1 or later: You can switch to inspect other browser windows or mobile devices that are open on your computer. Click CHOOSE in the AI Identification View window and select a different browser or device.

Next, add steps from this page or screen to your test.

Back to top

Run AI-based tests

After inspecting your application and creating test steps, run your AI-based test as you would run any other UFT One test. See Run / Debug tests.

You can run the same test on different operating systems and versions, as it is not based on implementation details.

Back to top

Convert technology-based tests to AI-based tests

The AI Transformation Assistant can help you convert property-based test steps in your existing tests to AI-based testing steps:

1. Enable the assistant.

   In Tools > Options > GUI Testing > AI, select Replace run results with AI object suggestions.

2. Run your tests.

   The test steps are performed on the application and the run report contains AI object suggestions instead of the results of your test run. Each step contains a suggestion for transforming the technology-based test step into an AI-based testing step.

3. Modify your test steps (AI conversion).

   Replace existing steps with the suggested AI object steps to create a more resilient and platform-agnostic test.

Back to top

Troubleshooting

AI text identification requires the Windows mediaserver.exe service to be running. Otherwise the following may occur:

• AI inspection cannot find objects By Text
• All FindText and FindTextBlock steps fail to return an AITextObject when running a test
• An error message indicates that an error occurred when calling the Media Server OCR service.

Solution: 

Open the Windows Services Manager and make sure the mediaserver.exe is running. Otherwise, start the service manually.

Back to top

See also:

• Improve AI-based test object identification

• AI-based testing: Supported control types

• The AI-based Testing section in the Object Model Reference