LLM protocol

The VuGen LLM protocol enables you to create scripts and load tests for applications that have an embedded large language model.

About the LLM protocol

The LLM protocol supports LLMs from providers such as OpenAI, Gemini, Anthropic, as well as custom models.

Note: Anthropic and custom models are supported from version 26.3.

Supported LLMs must meet the following requirements:

  • They return non-streaming responses only.

  • Responses are in JSON format.

  • Responses contain the number of tokens sent by the client, and the number of tokens received from the server.

LLM scripts contain standard Web - HTTP/HTML functions, with additional calls to a custom LLM function, llm_request. It is recommended that you add headers before the LLM requests, and use lr_eval_string to retrieve the body.

The llm_request function tracks the number of tokens sent to, and received from, the LLM. This data is used to calculate token processing rates (tokens per second), enabling performance analysis of your LLM interactions during load testing. LLM-specific monitors and graphs show the processing and decoding rates for LLM token input during a load test.

For examples of using the function with different LLM models, see llm_request function in the Function Reference.

Testing a custom LLM

When using the LLM function for a custom LLM , the AIModel must be set to custom, and two additional parameters are set: RequestTokens and ResponseTokens. These parameters define the JSON path that corresponds to the number of tokens sent in the request, and the number of tokens received from the server as a response.

For example, the following response is received from the server:

Copy code 
{
    "choices":
    [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "role": "assistant",
                "content":"Why don't scientists trust atoms?\\n\\nBecause they make up everything!",
                "reasoning_content":"Thinking Process:\\n\\n1. **Request:** The user wants a short joke.\\n2.  **Goal:*
                                    * Deliver a joke that is quick, punchy, and easily consumable.\\n3.  **Selection Strategy:
                                    ** Choose a classic, clean, and universally funny type of joke (e.g., knock-knock, pun,
                                    observational).\\n4.  **Drafting Options (Internal Check):**\\n    *   *Option A (Punny):* Why
                                    did the scarecrow win an award? (A bit too complex.)\\n    *   *Option B (Short setup/punchline):*
                                    The classic \\"two guys walking into a bar.\\" (Might be too long.)\\n **Option C (Animal/Simple):*
                                    Q: What do you call a fish with no eyes? (Short, simple, visual.)\\n5.  **Final Selection
                                    (Option C - Quick and clean):** This meets the \\"short\\" requirement perfectly.\\n\\n6.  
                                    **Output Generation.**"
                }
        }
    ],
    "created":1778157521,
    "model":"gemma-4-E2B-it-Q8_0.gguf",
    "object":"chat.completion",
    "usage":
    {
        "completion_tokens":228,
        "prompt_tokens":21,
        "total_tokens":249,
        "prompt_tokens_details":
        {
            "cached_tokens":0
        }
    },
    "id":"chatcmpl-xxxxx"
}

For the above response, the llm_request would look like this:

Copy code 
    llm_request("chat_completion",
        "URL= http://127.0.0.1:8080/v1/chat/completions",
        "AIModel=custom",
         Body={\"messages\": [{\"role\": \"user\", \"content\": \"give me a short joke\"}], \"stream\": false}",
        "RequestTokens=usage.prompt_tokens",
        "ResponseTokens=usage.completion_tokens",
        LAST);

See also