Evaluate GenAI responses

Use the Evaluate GenAI response step to assess the quality of an AI-generated response using configurable evaluation methods, metrics, and pass levels.

Evaluate GenAI response steps require an Aviator license.

In this topic:

Overview

Because AI responses are non-deterministic, validating their quality requires metric-based evaluation instead of exact matching.

Evaluate GenAI response steps let you customize the evaluation using multiple quality dimensions. For each metric, you set a minimum expected level on a scale from 1 (Fair) to 4 (Excellent). The step passes only if all selected metrics meet their configured minimum level.

To check whether a response meets specific conditions, use a Verify or Smart verify step. For details, see Add verification steps.

AI response location

Evaluate GenAI response steps can evaluate the following: 

• Anything displayed in the visible area of the application (AKA Current screen).

• A string stored in an input or output parameter.

Evaluation methods

The following evaluation methods are available:

Evaluation method	Purpose	Metrics
Evaluate compared to a baseline response	Compare the generated response against a baseline response that you provide.	Semantic Alignment, Key Element Coverage, Format Alignment, Avoidance of Deviations
Evaluate based on context you provide	Evaluate the quality of the response based on context you provide, such as the original prompt or additional background information.	Relevance and Accuracy, Completeness, Absence of Hallucination
General metrics	Assess the overall quality of the generated response without providing reference text.	Fluency & Grammar, Coherence, Bias & Fairness, Safety & Compliance

Create an Evaluate GenAI response step

Create the Evaluate step in the Editor and configure it in the step editing pane.

Prerequisite: Before the Evaluate step runs, the response you want to evaluate must be displayed in the application or stored in a parameter.

For example: In the steps prior to the Evaluate step, trigger AI generation in your application.

To evaluate GenAI response text in a parameter, store the text in an output parameter, as described in Store a value in an output parameter.

To create an Evaluate GenAI response step:

1. In the script Editor, click the Add step button and select Evaluate GenAI.

   The editor adds the step template. Alternatively, you can type the command manually.

   Syntax: Evaluate GenAI response in <TEST_SUBJECT>

2. In the step editing pane, in the Test subject list, select Parameter or Current screen.

3. Provide more details about the test subject:

   	For Current screen	For Parameter
   Required?	Optional	Required
   Where to specify	In the step editing pane	In the Editor or the step editing pane
   What to specify	In the Focus area, describe the area of the screen to evaluate. Note: The focus area is treated as a recommendation. After you run the script, the report shows the area of the screen that was actually evaluated.	The input or output parameter containing the string to evaluate.

4. In the step editing pane, configure one or more evaluation methods and metrics. For each method you configure, provide any required information or guidelines, and select one or more metrics. For details, see Configure evaluation methods.

5. (Optional) Configure the visual scope for this step. For details, see Visual scope.

6. If you want the test run on the current application to stop when the verification step fails, select Stop test run upon failure. An indicator is added to the step.

   If you didn't select the option, the step failure is reported but the run continues. (Default: Run does not stop)

After the script runs, the report for this step shows the evaluation status and metric results, including scores and explanations per metric.

Configure evaluation methods

Configure one or more evaluation methods for an Evaluate GenAI response step to use. This tells the script which evaluation criteria to use for your AI-generated response.

To configure evaluation methods:

1. In the step editing pane of an Evaluate GenAI response step, click an evaluation method.

   When selecting a method, keep in mind that each method uses different information for comparison and measures different metrics.

2. Depending on the method you configure, enter the following information:

   Evaluation method	Required information
   Evaluate compared to a baseline response	In the Baseline response field, enter the response to use for comparison. To reference a script parameter, use the @<parameter name> syntax.
   Evaluate based on context you provide	Provide at least one of the following: In The input provided to the AI in the application, enter the prompt or request used to generate the response. In Additional context, enter background information to use as context for the evaluation. To reference a script parameter, use the @<parameter name> syntax. Note: The evaluation step's context is comprised only of the parameter or application image and information you provide in this step. To include any information used earlier in the script, you must repeat it here.
   General metrics	No additional information required.

3. Select one or more evaluation metrics to check.

   If you configure an evaluation method but do not select any metrics, the details you configured are saved for future use, but the evaluation method is ignored during the script run.

4. For each selected metric, set a Minimum expected level from 1 (Fair) to 4 (Excellent). The step passes only if all selected metrics meet their configured minimum level.

5. (Optional) Add Additional guidelines to refine the evaluation, if the results were not satisfactory when you ran the script with the initial configuration.

   To reference a script parameter, use the @<parameter name> syntax.

6. Click Save. The evaluation method dialog box closes.

The step editing pane shows which metrics you selected and the minimum expected level configured for each metric.

To modify configured evaluation methods:

• To edit the details of a configured evaluation method, click the More options button and select Edit.

• To remove a specific metric from a configured method, click the x on the metric's chip.

• To completely remove all configuration details for an evaluation method, click the More options button and select Reset.

See also

• Manually edit steps
• Step syntax
• Add verification steps