LLM protocol

The VuGen LLM (large language model) protocol enables you to create scripts and load tests for applications that have an embedded LLM.

In this topic:

About the LLM protocol

The LLM protocol supports LLMs from providers such as OpenAI, Gemini, Anthropic, as well as custom models.

Supported LLMs must meet the following requirements:

• They return non-streaming responses only.

• Responses are in JSON format.

• Responses contain the number of tokens sent by the client, and the number of tokens received from the server.

LLM scripts contain standard Web - HTTP/HTML functions, with additional calls to a custom LLM function, llm_request. It is recommended that you add headers before the LLM requests, and use lr_eval_string to retrieve the body.

The llm_request function tracks the number of tokens sent to and received from the LLM. This data is used to calculate token processing rates (tokens per second), enabling performance analysis of your LLM interactions during load testing. LLM-specific monitors and graphs show the LLM token input processing and decoding rates during a load test.

For examples of using the function with different LLM models, see llm_request function in the Function Reference.

Testing a custom LLM

When using the LLM function for a custom LLM , the AIModel must be set to custom, and two additional parameters must be set: RequestTokens and ResponseTokens. These parameters define the JSON path that corresponds to the number of tokens sent in the request and the number of tokens received from the server as a response.

For example, if the following response is received from the server:

{
    "choices":
    [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "role": "assistant",
                "content":"Why don't scientists trust atoms?\\n\\nBecause they make up everything!",
                "reasoning_content":"Thinking Process:\\n\\n1. **Request:** The user wants a short joke.\\n2.  **Goal:*
                                    * Deliver a joke that is quick, punchy, and easily consumable.\\n3.  **Selection Strategy:
                                    ** Choose a classic, clean, and universally funny type of joke (e.g., knock-knock, pun,
                                    observational).\\n4.  **Drafting Options (Internal Check):**\\n    *   *Option A (Punny):* Why
                                    did the scarecrow win an award? (A bit too complex.)\\n    *   *Option B (Short setup/punchline):*
                                    The classic \\"two guys walking into a bar.\\" (Might be too long.)\\n **Option C (Animal/Simple):*
                                    Q: What do you call a fish with no eyes? (Short, simple, visual.)\\n5.  **Final Selection
                                    (Option C - Quick and clean):** This meets the \\"short\\" requirement perfectly.\\n\\n6.  
                                    **Output Generation.**"
                }
        }
    ],
    "created":1778157521,
    "model":"gemma-4-E2B-it-Q8_0.gguf",
    "object":"chat.completion",
    "usage":
    {
        "completion_tokens":228,
        "prompt_tokens":21,
        "total_tokens":249,
        "prompt_tokens_details":
        {
            "cached_tokens":0
        }
    },
    "id":"chatcmpl-xxxxx"
}

For the above response, the llm_request would look like this:

    llm_request("chat_completion",
        "URL= http://127.0.0.1:8080/v1/chat/completions",
        "AIModel=custom",
         Body={\"messages\": [{\"role\": \"user\", \"content\": \"give me a short joke\"}], \"stream\": false}",
        "RequestTokens=usage.prompt_tokens",
        "ResponseTokens=usage.completion_tokens",
        LAST);

See also

• llm_request function
• Web - HTTP/HTML protocol