AI-based testing

Mobile, web, and windows-based SAP GUI tests

This topic explains how to use OpenText Functional Testing's Artificial Intelligence (AI) Features in your tests to identify objects the way a person would. This enables you to run the same test on different platforms and versions, regardless of the objects' implementation.

AI-based testing overview

OpenText Functional Testing's AI Object Detection enables your tests to interact with the application you are testing in the same way a person would. OpenText Functional Testing uses AI to identify objects visually, based on a wide variety of images, context, and sometimes text.

For example, OpenText Functional Testing AI can identify many types of search fields, user profile areas, input fields, buttons, and shopping carts.

Some advantages of AI-based object identification are: 

Easier to edit

Test scripts are more intuitive.
Test multiple environments

Tests are technology agnostic, identifying objects visually, regardless of the technology details used behind the scenes.
Test resilience

Tests are easier to maintain, as an object changing location, framework, or even shape, won’t break the test script as long as the object remains visually similar or its purpose remains clear.

AI Object-Detection service

The AI Object-Detection service that provides OpenText Functional Testing's AI Object Detection is computationally intensive. We recommend that you use a powerful computer to benefit from AI Object Detection and achieve optimal performance. For recommended system requirements, see Support Matrix.

OpenText Functional Testing's AI Object-Detection service can run on, and be consumed from, different sources. The service can run locally on your OpenText Functional Testing computer, remotely, on a designated machine in your network, or on the cloud. The local AI Object-Detection service is used by default.

You can choose to use the remote AI Object-Detection service or the AI Object-Detection Cloud Service, depending on your needs:

  • Use the remote AI Object-Detection service on one powerful computer, letting it provide AI capabilities for multiple weaker OpenText Functional Testing computers. For details, see Use the remote AI Object-Detection service.

  • Use the AI Object-Detection Cloud Service to reduce local use of resources and get the most up-to-date AI support. For details, see Use the AI Object-Detection Cloud Service.

    Note: The AI Object-Detection Cloud Service is available in OpenText Functional Testing as a Technical Preview upon request. It is not available in OpenText Functional Testing by default. To learn more, contact your account team and send an email to AIODCloudService@opentext.com.

AI Object Model

OpenText Functional Testing's AI Object Detection uses incorporated AI Object Models to assist OpenText Functional Testing in learning your application and identifying test objects.

Updated AI Object Models can improve performance, accuracy, and identification capabilities. When new models are available, you can download them and use them for your AI-based testing. For details, see Update local AI Object Models.

Back to top

Prerequisites

To use OpenText Functional Testing's AI Object Detection, it must be installed and enabled. By default, OpenText Functional Testing's installation is set up to install and enable OpenText Functional Testing's AI Object Detection.

If AI Object Detection is not installed (the Options dialog box does not display the GUI Testing > AI Object Detection pane), do one of the following: 

  • Run the OpenText Functional Testing installation, select Modify, and choose AI Object Detection option. See Custom Setup screen.

  • Run the OpenText Functional Testing silent installation and use the ADDLOCAL parameter to enable AI Object Detection. See Include specific features.

To enable or disable AI Object Detection after installation, use the Tools > Options > GUI Testing > AI Object Detection > Enable AI Object Detection option.

Prepare your test solution and create a test for a web, mobile, or SAP GUI for Windows application.

To start identifying AI objects to use in your test, continue with one of the following tasks:

Back to top

Inspect your application for objects

Use AI object inspection to identify objects in your application that you can use in your test steps.

The instructions below describe how to prepare your application for testing, how to use AI to identify objects in your application, and how to send OpenText feedback on the identification, if you would like to.

Prerequisites

If you are testing an application on a mobile device

  1. Make sure the Mobile Add-in is installed and loaded.

    To test web applications on your mobile device, make sure the Web Add-in is also installed and loaded.

  2. Connect to Digital Lab and open your mobile device and the application you want to test. For details, see Open the remote access window.

If you are testing a desktop web application

  1. Make sure the Web Add-in is installed and loaded.

  2. Make sure that your browser is set up with the OpenText UFT Agent extension. For details, see Test Web apps on specific browsers.

  3. Open the application.

    For best identification results:

    • Maximize the browser window and set the zoom level to 100%.

    • Set your Windows display settings to display all text and apps at 100%.

    • Make sure no banners are displayed in the browser aside from your web page.

Supported browsers: Chrome, Edge, Firefox, Internet Explorer.

Headless Chrome is supported as well. See Prepare to run tests on Headless Chrome.

Other headless browsers are not supported.

If you are testing an SAP GUI for Windows application

  1. Make sure that you perform the following tasks as described in Enable support for SAP GUI for Windows.

    1. Install SAP GUI Scripting.

    2. Enable SAP GUI Scripting on the server side.

    3. Enable SAP GUI Scripting on the client side.

  2. Make sure the SAP Solutions Add-in is installed and loaded.

  3. Open the application.

    For best identification results, set your Windows display settings to display all text and apps at 100%.

Note: AI-based testing for SAP GUI for Windows applications is officially supported on the Classic and Signature themes.

To identify single objects

In the Object Spy or Object Identification Center, as you spy on objects in mobile, web, or SAP GUI for Windows applications, OpenText Functional Testing also presents the AI object you could use to represent the same object.

When spying on a single object When you click on an object in the application, an AI object is suggested only for the object you clicked. AI objects are not suggested for any other object in the hierarchy displayed in the Spy.

When spying on multiple objects using the OIC

AI object suggestions are available in the Edit view of each spied object. To open the edit view, hover over an object in the list of spied objects and click the Edit button.

To identify all objects

  1. Open the AI Object Inspection window to inspect your application and detect all AI objects in it.

    To open the window, do one of the following: 

    • Select AI > AI Object Inspection.

    • Click the AI Object Inspection toolbar button.

    • In the Object Spy or Object Identification Center, click INSPECT.

      Tip: This button is available when the AI Auto Inspection option is enabled.

  2. Click a web application, an SAP GUI for Windows application, or the remote access window displaying your mobile application. If you click an unsupported context or press Esc, the Select Application button appears to prompt you to click again.

    Note: You can skip this step if you began the inspection with a web Browser, mobile Device, or SAPGUISession test object as the top-level object in your spied hierarchy. OpenText Functional Testing automatically selects that test object for inspection.

    The Live Application tab displays the current screen of the application, highlighting all of the detected objects.

  3. Decide whether to show Visual Elements, Texts, or both, to see either the objects that OpenText Functional Testing detected visually, or areas of text in the application, or both.

  4. (Tech preview) If your application contains container objects such as tables and you want to inspect the lower-level objects inside the tables, turn on the Containers switch to initiate a reinspection. (The switch name may vary between versions.)

    Note: Table cell objects are identified using ABBYY OCR.

    To identify TableCell objects, you must have the ABBYY OCR Engine feature installed in OpenText Functional Testing.

  5. After identifying the AI objects, you can Add AI-based steps to your tests.

Help design the future of AI-based testing in OpenText Functional Testing

Click How is the detection? Help us to improve to open the Feedback Tool and send OpenText feedback about the object detection. For details, see Send AI Object Identification feedback.

Back to top

Inspect application mockups for objects

Use AI Object Mockup Inspection to inspect application mockups and identify objects to use in your test. This enables you to design and prepare your test even before your application is fully developed.

Prerequisites

AI Object Mockup Inspection does not require a ready application. Instead, create a local folder containing images in .jpg, .jpeg, or .png format.

Inspect application mockups

  1. In the OpenText Functional Testing title bar, click AI > AI Object Mockup Inspection.

  2. In the MOCKUP IMAGES tab of the AI Object Inspection window, select the inspection context and click Browse Folder to select a folder containing images.

    Inspection contexts: Non-Mobile/Mobile

    The AI Object Inspection window inspects the image that appears first in the file name order and highlights all identified visual elements by default.

  3. Decide whether to show Visual Elements, Texts, or both, to see either the objects that OpenText Functional Testing detected visually, or areas of text in the application, or both.

  4. (Tech preview) If your application contains container objects such as tables and you want to inspect the lower-level objects inside the tables, turn on the Containers switch to initiate a reinspection. (The switch name may vary between versions.)

    Note: Table cell objects are identified using ABBYY OCR.

    To identify TableCell objects, you must have the ABBYY OCR Engine feature installed in OpenText Functional Testing.

  5. Decide whether to change the AI Object Inspection context type by selecting Non-Mobile or Mobile.

  6. Click the down arrow near the folder icon to select a different folder, synchronize with the current folder, or open the current folder.

  7. To view all images in your folder, click the down arrow on the top to show the image gallery. In the gallery, you can:

    • Navigate between images in the folder with arrows.

    • Click the Search button to search for a specific image.

    • Select whether to view the images in Grid View or Line View.

    • Select Name or Date to sort images by the file name or modified time.

  8. After identifying the AI objects, you can Add AI-based steps to your tests.

Help design the future of AI-based testing in OpenText Functional Testing

Click How is the detection? Help us to improve to open the Feedback Tool and send OpenText feedback about the object identification. For details, see Send AI Object Identification feedback.

Back to top

Add AI-based steps to your tests

To add AI-based steps in your tests:

  • Set the AI context

  • Add test steps

  • Add checkpoints to your test (optional)

For details, see Add AI-based steps to your tests.

Back to top

Inspect the next application page/screen

When you finish creating test steps for one page or screen in your application and you want to continue on another one, follow the steps below:

To inspect the next application page/screen

  1. Click the down arrow near Context , select Go to inspected application , and navigate to the desired location in the application.

  2. In the AI Object Inspection window, click Re-inspect to load the new application page or screen and reinspect it.

    • If you don't reload and try to run a step from the AI Object Inspection window, OpenText Functional Testing runs the step based on the previous page’s inspection, resulting in an error or performing the operation on the new page.

    • If multiple remote access windows or browser windows are open, the inspection session interacts with only one.

    • You can switch to inspect other browser windows or mobile devices that are open on your computer. Click the down arrow near Context and then select Select a different application in the AI Object Inspection window.

    • If you need to perform steps on the application to prepare it for inspection, you can use the Delayed re-inspect.

      Click the down arrow near Delayed re-inspect and set the required delay.

      Click Go to begin the countdown, open the application, and perform steps such as hover or menu clicks to bring the application to the state you want to inspect.

      When the delay timer expires, the application is re-inspected.

  3. Add steps from the new page or screen to your test.

Tip: You may need to set the delay to a larger value if you have a slow network.

Back to top

Run AI-based tests

After inspecting your application and creating test steps, run your AI-based test as you would run any other OpenText Functional Testing test. See Run / Debug tests.

You can run the same test on different operating systems and versions because it is not based on implementation details.

Back to top

Convert technology-based tests to AI-based tests

The AI Transformation Assistant can help you convert property-based test steps in your existing tests to AI-based testing steps:

  1. Enable the assistant.

    In Tools > Options > GUI Testing > AI > General, select Replace run results with AI object suggestions.

  2. Run your tests.

    The test steps are performed on the application and the run report contains AI object suggestions instead of the results of your test run. Each step contains a suggestion for transforming the technology-based test step into an AI-based testing step.

    Note: The AI Transformation Assistant does not check the object hierarchy inside table or calendar objects, and does not suggest AITable or AICalendar containers.

  3. Modify your test steps (AI conversion).

    Replace existing steps with the suggested AI object steps to create a more resilient and platform-agnostic test.

Back to top

Troubleshooting

AI text identification requires the Windows mediaserver.exe service to be running. Otherwise the following may occur:

  • AI object inspection cannot find objects By Text.
  • All FindText and FindTextBlock steps fail to return an AITextObject when running a test.
  • An error message indicates that an error occurred when calling the Media Server OCR service.

Solution: 

Open the Windows Services Manager and make sure the mediaserver.exe is running. Otherwise, start the service manually.

Back to top

See also: