Tips and tricks for AI-based testing

Mobile, web, and windows-based SAP GUI tests

This topic provides tips and tricks for writing AI-based tests in OpenText Functional Testing.

Include the SetContext step

When you first start the Inspection and select an application, such as a web page, a SetContext step is suggested. Make sure to click Add to test to add that step to your action before any other AIUtil steps. That SetContext step tells the AI engine what to interact with.

The context may change when you switch to another tab, or when an application dialog box opens. Use Inspection to create the SetContext steps to add to your test for these cases.

Back to top

Object identification tips

This section lists some common object identification suggestions. For additional ideas, see Improve AI-based test object identification.

Delayed re-inspect

If you have pop-ups or menus that appear only when you hover over the item, use the delayed re-inspect option. Initiate the delayed re-inspection, then hover over an item to display the control you want to capture. For details, see AI-based testing.

RegisterCustomClass

If your application includes a control type that is not supported by AI object identification, you can register a custom class to represent it. For details, see Describe a control using an image.

Use VRI

To help identify an object uniquely and consistently, describe the object based on other objects in its proximity. For details, see Identify objects by relative location.

Varying text strings

To identify text in the application that changes from time to time, such as price, tax, or total, consider the following: 

  • Use regular expressions and base your identification on the permanent parts or formats of the text string. For details, see Use regular expressions.

  • Use FindText instead of FindTextBlock for the object's text description. FindText searches for a string or substring, while FindTextBlock looks for a match with the full text of the control. For details, see FindText, and FindTextBlock in the Object Model Reference for GUI Testing.

Back to top

Text recognition in AI-based testing

Keep the following in mind when using text to identify AI objects.

  • If the AI OCR doesn't accurately recognize the text in your application, you can switch to the UFT OCR, and customize that OCR for your application. This can be done as a setting in options, or with a script command at run time. For details, see Configure text recognition options. When AI OCR is accurate, use it to leverage its faster performance.

  • Inspection identifies all the objects displayed in the application, while a test run searches for a specified object. Keep in mind that text may be identified differently in each of this scenarios.

    For example:

    • Sometimes the OCR recognizes a number as a letter during inspection, but a test step describing the object with the number still runs successfully. This is because the AI-based mechanism still identifies the text, just with a slightly lower confidence level.

    • When your recordings have somewhat unexpected text like "PA SSWO rd", it might be that the font was slightly misinterpreted. However you can change that text to the more expected text like "Password" and the step should run successfully.

  • Prevent OCR failure on misspelled text. When running tests, OCR fails to identify text displayed with a misspelling indication, such as a wavy line under the text.

    Use one of the following solutions:

    • Deactivate spell checking in your browser or application.

    • Move the focus away from the field with the misspelled text. To do this, add a step to your test that clicks somewhere outside of the text field before the step that retrieves the text.

Back to top

Performance

The following conditions can affect the performance of AI-based tests: 

Improve performance by minimizing re-inspections during a test run

Each AIUtil command performs a full scan of the application context. If you know the application is not expected to change between steps, use the AIUtil.Context.Freeze and UnFreeze methods to prevent re-inspection.

Use a GPU

AI Object Detection is computationally intensive. Using a GPU machine to run the AI Object-Detection service will help it run faster. However, depending on your requirements, the performance achieved without a GPU may be sufficient.

Use the Verify object identification option judiciously

Verifying object identification before running the step slows down the test runs. You can turn the verification off, but then you must make sure that the application is finished changing before you continue to the next step. Verification helps, for example, with objects that fly in to the app, as it determines when the object's location is final. If some of your application is dynamic and other parts are static, you might want to turn verification off and on dynamically. For details, see Verify object identification.

Back to top

Learn how to transform your property-based test steps to AI-based ones

Run your tests with the AI Transformation Assistant enabled, to receive suggestions for transforming technology-based test steps to AI-based testing steps. Replace existing steps with the suggested AI object steps to create a more resilient and platform-agnostic test.

To enable the assistant in Tools > Options > GUI Testing > AI, select Replace run results with AI step suggestions.

When this option is selected, your test run results do not reflect the results of the test. Instead, the report suggests an AI object and operation for any step where a property-based test object can be replaced with an AI object.

Note: The AI Transformation Assistant does not provide AITable or AICalendar object suggestions.

Back to top

Send keyboard commands to AI objects

In some applications, a Type operation on an AI object requires an additional keyboard command, such as pressing Enter. You can simulate this command using Windows Scripting or Device Replay for desktop browsers and Device.EnterKeys for mobile applications.

Example #1 Type a model name in the Cell Phone combo box and send the Enter key to trigger selection

  • For desktop browsers

    • Using Windows Scripting

      Copy code
      AIUtil("combobox", "Cell Phone").Type "Pixel 2"
      Dim mySendKeys
      Set mySendKeys = CreateObject("WScript.shell")
      'Send Enter key
      mySendKeys.SendKeys("~")       

      For details on the SendKeys Method in Windows Script Host, see the Microsoft VBScript Reference.

    • Using Device Replay

      Copy code
      AIUtil("combobox", "Cell Phone").Type "Pixel 2"
      wait 5    
      Dim myDeviceReplay
      Set myDeviceReplay = CreateObject("Mercury.DeviceReplay")

      myDeviceReplay.PressKey 28            'Send Enter key

      ' Other example keys:
      ' myDeviceReplay.PressKey 14           'Send Backspace
      ' myDeviceReplay.PressKey 211          'Send Delete
      ' myDeviceReplay.PressKey 15           'Send Tab
      ' myDeviceReplay.PressKey 199          'Send Home

      For details, see DeviceReplay Object in the Object Model Reference for GUI Testing.

  • For mobile applications

    Using Device.EnterKeys

    Copy code
    AIUtil("combobox", "Cell Phone").Type "Pixel 2"
    wait 5    
    Device("Device").Enterkeys typeKey, "Enter"

    Tip: Add a wait step before sending keys to the application if the mobile device is slow to process keyboard input.

    For details, see EnterKeys Method of Device Object.

Example #2 Clear a text box before writing in it on desktop browsers

Using Device Replay

Copy code
Sub CleanInputField(Label)
      AIUtil(input,Label).Click
    Dim myDeviceReplay
    Set myDeviceReplay = CreateObject("Mercury.DeviceReplay")
      for i = 1 to 25
            myDeviceReplay.PressKey 14
      next 
end sub

Back to top

Run-result screen captures during context freeze

After a AIUtil.Context.Freeze step, run-result screen captures for AI-based steps reflect the frozen context. Changes that occur in the application are not captured.

To capture changes that occur in the application, do one of the following: 

  • Unfreeze the context.

  • Add a CaptureBitmap step on a top-level object, such as Browser or Device. Then add the bitmap to the results, using Reporter.ReportHtmlEvent.

  • If you configured saving screen captures to the results for each step, add a Wait or Sync step on a top-level object, such as Browser or Device.

  • Configure run results to include a movie.

Back to top

Handle cases where AIUtil("search").Search does not perform a search

In some applications, the search box is shown after clicking the search icon. In these cases, AIUtil("search").Search may not be able to perform a search operation in your application.

You can perform the search using multiple steps, as demonstrated in the following examples:

Scenario #1

Text is not typed in the search box after AIUtil("search").Search

Suggested solution

Copy code
AIUtil("search").Click
'Wait a while until the search box is displayed
wait 2
AIUtil("search").Search "telekom"

Scenario #2

After clicking the search icon, the search box is not in focus and two search icons are displayed

Suggested solution

Copy code
AIUtil("search").Click
'Wait a while until the search box is displayed
wait 2
AIUtil("search",micNoText,micFromBottom,1).Search "telekom"

Scenario #3

After clicking the search icon, the search icon disappears and the search box is not in focus

Suggested solution

Copy code
AIUtil("search").Click
'Wait a while until the search box is displayed
wait 2
AIUtil("input","search").Type "telekom"

'Press Enter key
Dim myDeviceReplay
Set myDeviceReplay = CreateObject("Mercury.DeviceReplay")
myDeviceReplay.PressKey 28

Back to top

Find an object in a scrollable pane

This example demonstrates how to use the AIUtil.Scroll method to find objects in scrollable panes of your application.

Copy code
Set ObjBrowser= Browser("creationtime:=1")
ObjBrowser.Maximize
AIUtil.SetContext ObjBrowser
'---------------------------------------------------------------------------------------
ObjBrowser.Navigate https://www.youtube.com/
ObjBrowser.Sync
'wait(2)
'---------------------------------------------------------------------------------------
print ClickAndScrollTo ( AIUtil("home"), AIUtil.FindTextBlock("Report history"))
'---------------------------------------------------------------------------------------
ObjBrowser.Navigate https://usa.banggood.com
ObjBrowser.Sync
'wait(2)
-----------------------------------------------------------------------------------------
print ClickAndScrollTo (AIUtil("down_triangle", micAnyText, micFromTop, 5), AIUtil.FindTextBlock("Home Appliances", micFromBottom, 1))
'--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Function ClickAndScrollTo (ObjStart, ObjTarget)
    ClickAndScrollTo=false
    AIUtil.RunSettings.AutoScroll.Disable
    ObjStart.Highlight   
    ObjStart.Click
    For i = 1 To 4
        if ObjTarget.Exist(0) then
        ObjTarget.Highlight
        ObjTarget.hover
        ClickAndScrollTo=true
               exit function 
        end if
'ScrollUsingPointerPosition([in] micAIScrollDirection direction, [in] double numberOfWheelTicks);
'Move the mouse over the scroll object using AIUtil.Hover or another method, for example device replay 
        AIUtil.Scroll "down", 1
    Next
End Function

Back to top

Horizontal scrolling on an SAP GUI application

If AIUtil.Scroll and AIUtil.ScrollOnObject do not provide horizontal scrolling on an SAP GUI application, you can use AIUtil.Click and AIUtil.MultiClick to perform horizontal scrolling.

Copy code
' Click the right triangle to perform right scrolling
' If there are more than one right_triangles, you can identify the control by relative location.
Set ObjRightTriangle = AIUtil("right_triangle")
'Click the right triangle four times
ObjRightTriangle.MultiClick 4
' Click the left triangle to perform left scrolling
Set ObjLeftTriangle = AIUtil("left_triangle")
'Click the left triangle
ObjLeftTriangle.Click

Back to top

See also: