Tips and tricks for AI-based testing
Mobile, web, and windows-based SAP GUI tests
This topic provides tips and tricks for writing AI-based tests in OpenText Functional Testing.
Include the SetContext step
When you first start the Inspection and select an application, such as a web page, a SetContext step is suggested. Make sure to click Add to test to add that step to your action before any other AIUtil steps. That SetContext step tells the AI engine what to interact with.
The context may change when you switch to another tab, or when an application dialog box opens. Use Inspection to create the SetContext steps to add to your test for these cases.
Object identification tips
This section lists some common object identification suggestions. For additional ideas, see Improve AI-based test object identification.
Delayed re-inspect
If you have pop-ups or menus that appear only when you hover over the item, use the delayed re-inspect option. Initiate the delayed re-inspection, then hover over an item to display the control you want to capture. For details, see AI-based testing.
RegisterCustomClass
If your application includes a control type that is not supported by AI object identification, you can register a custom class to represent it. For details, see Describe a control using an image.
Use VRI
To help identify an object uniquely and consistently, describe the object based on other objects in its proximity. For details, see Identify objects by relative location.
Varying text strings
To identify text in the application that changes from time to time, such as price, tax, or total, consider the following:
-
Use regular expressions and base your identification on the permanent parts or formats of the text string. For details, see Use regular expressions.
-
Use FindText instead of FindTextBlock for the object's text description. FindText searches for a string or substring, while FindTextBlock looks for a match with the full text of the control. For details, see FindText, and FindTextBlock in the Object Model Reference for GUI Testing.
Text recognition in AI-based testing
Keep the following in mind when using text to identify AI objects.
-
If the AI OCR doesn't accurately recognize the text in your application, you can switch to the UFT OCR, and customize that OCR for your application. This can be done as a setting in options, or with a script command at run time. For details, see Configure text recognition options. When AI OCR is accurate, use it to leverage its faster performance.
-
Inspection identifies all the objects displayed in the application, while a test run searches for a specified object. Keep in mind that text may be identified differently in each of this scenarios.
For example:
-
Sometimes the OCR recognizes a number as a letter during inspection, but a test step describing the object with the number still runs successfully. This is because the AI-based mechanism still identifies the text, just with a slightly lower confidence level.
-
When your recordings have somewhat unexpected text like "PA SSWO rd", it might be that the font was slightly misinterpreted. However you can change that text to the more expected text like "Password" and the step should run successfully.
-
-
Prevent OCR failure on misspelled text. When running tests, OCR fails to identify text displayed with a misspelling indication, such as a wavy line under the text.
Use one of the following solutions:
-
Deactivate spell checking in your browser or application.
-
Move the focus away from the field with the misspelled text. To do this, add a step to your test that clicks somewhere outside of the text field before the step that retrieves the text.
-
Performance
The following conditions can affect the performance of AI-based tests:
Improve performance by minimizing re-inspections during a test run
Each AIUtil command performs a full scan of the application context. If you know the application is not expected to change between steps, use the AIUtil.Context.Freeze and UnFreeze methods to prevent re-inspection.
Use a GPU
AI Object Detection is computationally intensive. Using a GPU machine to run the AI Object-Detection service will help it run faster. However, depending on your requirements, the performance achieved without a GPU may be sufficient.
Use the Verify object identification option judiciously
Verifying object identification before running the step slows down the test runs. You can turn the verification off, but then you must make sure that the application is finished changing before you continue to the next step. Verification helps, for example, with objects that fly in to the app, as it determines when the object's location is final. If some of your application is dynamic and other parts are static, you might want to turn verification off and on dynamically. For details, see Verify object identification.
Learn how to transform your property-based test steps to AI-based ones
Run your tests with the AI Transformation Assistant enabled, to receive suggestions for transforming technology-based test steps to AI-based testing steps. Replace existing steps with the suggested AI object steps to create a more resilient and platform-agnostic test.
To enable the assistant in Tools > Options > GUI Testing > AI, select Replace run results with AI step suggestions.
When this option is selected, your test run results do not reflect the results of the test. Instead, the report suggests an AI object and operation for any step where a property-based test object can be replaced with an AI object.
Note: The AI Transformation Assistant does not provide AITable or AICalendar object suggestions.
Send keyboard commands to AI objects
In some applications, a Type operation on an AI object requires an additional keyboard command, such as pressing Enter. You can simulate this command using Windows Scripting or Device Replay for desktop browsers and Device.EnterKeys for mobile applications.
Example #1 Type a model name in the Cell Phone combo box and send the Enter key to trigger selection
-
For desktop browsers
-
Using Windows Scripting
Copy codeAIUtil("combobox", "Cell Phone").Type "Pixel 2"
Dim mySendKeys
Set mySendKeys = CreateObject("WScript.shell")
'Send Enter key
mySendKeys.SendKeys("~")For details on the SendKeys Method in Windows Script Host, see the Microsoft VBScript Reference.
-
Using Device Replay
Copy codeAIUtil("combobox", "Cell Phone").Type "Pixel 2"
wait 5
Dim myDeviceReplay
Set myDeviceReplay = CreateObject("Mercury.DeviceReplay")
myDeviceReplay.PressKey 28 'Send Enter key
' Other example keys:
' myDeviceReplay.PressKey 14 'Send Backspace
' myDeviceReplay.PressKey 211 'Send Delete
' myDeviceReplay.PressKey 15 'Send Tab
' myDeviceReplay.PressKey 199 'Send HomeFor details, see DeviceReplay Object in the Object Model Reference for GUI Testing.
-
-
For mobile applications
Using Device.EnterKeys
Copy codeAIUtil("combobox", "Cell Phone").Type "Pixel 2"
wait 5
Device("Device").Enterkeys typeKey, "Enter"Tip: Add a wait step before sending keys to the application if the mobile device is slow to process keyboard input.
For details, see EnterKeys Method of Device Object.
Example #2 Clear a text box before writing in it on desktop browsers
Using Device Replay
Sub CleanInputField(Label)
AIUtil(input,Label).Click
Dim myDeviceReplay
Set myDeviceReplay = CreateObject("Mercury.DeviceReplay")
for i = 1 to 25
myDeviceReplay.PressKey 14
next
end sub
Run-result screen captures during context freeze
After a AIUtil.Context.Freeze step, run-result screen captures for AI-based steps reflect the frozen context. Changes that occur in the application are not captured.
To capture changes that occur in the application, do one of the following:
-
Unfreeze the context.
-
Add a CaptureBitmap step on a top-level object, such as Browser or Device. Then add the bitmap to the results, using Reporter.ReportHtmlEvent.
-
If you configured saving screen captures to the results for each step, add a Wait or Sync step on a top-level object, such as Browser or Device.
-
Configure run results to include a movie.
Handle cases where AIUtil("search").Search does not perform a search
In some applications, the search box is shown after clicking the search icon. In these cases, AIUtil("search").Search may not be able to perform a search operation in your application.
You can perform the search using multiple steps, as demonstrated in the following examples:
Scenario #1
Text is not typed in the search box after AIUtil("search").Search
Suggested solution
AIUtil("search").Click
'Wait a while until the search box is displayed
wait 2
AIUtil("search").Search "telekom"
Scenario #2
After clicking the search icon, the search box is not in focus and two search icons are displayed
Suggested solution
AIUtil("search").Click
'Wait a while until the search box is displayed
wait 2
AIUtil("search",micNoText,micFromBottom,1).Search "telekom"
Scenario #3
After clicking the search icon, the search icon disappears and the search box is not in focus
Suggested solution
AIUtil("search").Click
'Wait a while until the search box is displayed
wait 2
AIUtil("input","search").Type "telekom"
'Press Enter key
Dim myDeviceReplay
Set myDeviceReplay = CreateObject("Mercury.DeviceReplay")
myDeviceReplay.PressKey 28
Find an object in a scrollable pane
This example demonstrates how to use the AIUtil.Scroll method to find objects in scrollable panes of your application.
Set ObjBrowser= Browser("creationtime:=1")
ObjBrowser.Maximize
AIUtil.SetContext ObjBrowser
'---------------------------------------------------------------------------------------
ObjBrowser.Navigate https://www.youtube.com/
ObjBrowser.Sync
'wait(2)
'---------------------------------------------------------------------------------------
print ClickAndScrollTo ( AIUtil("home"), AIUtil.FindTextBlock("Report history"))
'---------------------------------------------------------------------------------------
ObjBrowser.Navigate https://usa.banggood.com
ObjBrowser.Sync
'wait(2)
-----------------------------------------------------------------------------------------
print ClickAndScrollTo (AIUtil("down_triangle", micAnyText, micFromTop, 5), AIUtil.FindTextBlock("Home Appliances", micFromBottom, 1))
'--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Function ClickAndScrollTo (ObjStart, ObjTarget)
ClickAndScrollTo=false
AIUtil.RunSettings.AutoScroll.Disable
ObjStart.Highlight
ObjStart.Click
For i = 1 To 4
if ObjTarget.Exist(0) then
ObjTarget.Highlight
ObjTarget.hover
ClickAndScrollTo=true
exit function
end if
'ScrollUsingPointerPosition([in] micAIScrollDirection direction, [in] double numberOfWheelTicks);
'Move the mouse over the scroll object using AIUtil.Hover or another method, for example device replay
AIUtil.Scroll "down", 1
Next
End Function
Horizontal scrolling on an SAP GUI application
If AIUtil.Scroll and AIUtil.ScrollOnObject do not provide horizontal scrolling on an SAP GUI application, you can use AIUtil.Click and AIUtil.MultiClick to perform horizontal scrolling.
' Click the right triangle to perform right scrolling
' If there are more than one right_triangles, you can identify the control by relative location.
Set ObjRightTriangle = AIUtil("right_triangle")
'Click the right triangle four times
ObjRightTriangle.MultiClick 4
' Click the left triangle to perform left scrolling
Set ObjLeftTriangle = AIUtil("left_triangle")
'Click the left triangle
ObjLeftTriangle.Click
See also: