Share this page

# Kubernetes configuration settings

This section describes how to configure settings for a Kubernetes image on the LoadRunner Enterprise server.

In this topic:

## Configure Kubernetes settings

When using a Kubernetes orchestrator with Dockerized host images, you can configure the following Kubernetes settings in the kubernetes.config file:

- Select the Kubernetes image pull policy.

- Define how Kubernetes containerized load generators are accessed. You can now connect to pods (containerized load generators) via a node port or a load balancer using an external IP.

- Set timeout to get load balancer external IPs and a delay for connecting the Controller to the pods.

- Use annotations to attach non-identifying metadata to Kubernetes services.

- Disable the service account auto-mount in Kubernetes.

- Enable Kubernetes health probes.

- Disable TLS verification for requests sent to a Kubernetes cluster.

Note:

- Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service (EKS) load balancers with External IPs are currently supported. For supported versions, see the Supported Protocols guide available from the Support Matrix.

- We recommend making a backup of the kubernetes.config file because the file is deleted during uninstall or upgrade.

Back to top

## Configure the image pull policy

When creating the pod, you can specify the image pull policy, which guides the kubelet service on how to pull the specified image during an update. Kubernetes has three image pull policies: IfNotPresent, Always, and Never.

To configure the image pull policy:

1. Open the kubernetes.config file located in the <LRE server installation path>\dat folder.

2. Under <appsettings>, add a key named "ImagePullPolicy" with one of the following values:

   AKS | EKS |
   --- | --- | ---
   "Pull If Not Present" | "IfNotPresent" | Kubernetes only pulls the image when it does not already exist on the node. This is the default value if no value is set, or if the pull policy key is missing from the Kubernetes configuration file.
   "Pull Always" | "Always" | Kubernetes always pull the image from the registry when updating the container.
   "Pull Never" | "Never" | Kubernetes never pulls the image.

   Copy code
   Example of configuring the Kubernetes image pull policy for EKS:

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="ImagePullPolicy" value="Always" />
         </appSettings>
       </configuration>

3. If you are using a Kubernetes cluster with a private registry that requires authorization, you also need to perform the following:

   1. In Docker Hub, log in to your private registry by running `docker login`

   2. Create a secret of your private registry to pull a private image:

      Copy code

          kubectl create secret generic <secret name> \
             --from-file=.dockerconfigjson=<path/to/.docker/config.json> \
             --type=kubernetes.io/dockerconfigjson

   3. Keep a record of your registry secret. You need to add it when configuring the orchestrator settings in LoadRunner Enterprise Administration (Orchestration > Secret Name box). For details, see Kubernetes configuration settings.

4. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

## Define how containerized load generators are accessed

You can define how the Kubernetes pods, or containerized load generators, are accessed based on your cluster configuration. If you use a load balancer such as AKS, you must use the cloud provider's load balancer. If you are using Kubernetes on-premises, you can connect to pods via a node port or a load balancer using an external IP. Node means the machine which runs the pods or containers.

Example: This diagram shows the implementation of the Load balancer service with external IP flow:

To define how containerized load generators are accessed:

1. Open the kubernetes.config file in the <LRE server installation path>\dat folder.

2. Under <appsettings>, add a key named "KubernetesAccessType" with one of the following values:

   - "NodePortNodeNames". Exposes the service on an internal IP in the cluster. This is the default value if no value is set, or if the access type key is missing from the Kubernetes configuration file.

   - "NodePortExternalIPs". Exposes services to external clients via a static port on each node's IP.

   - "LoadBalancerIngress". Exposes the service to external clients via the cloud provider's load balancer. This enables provisioning load generators that are set up in different networks to the Controller. When using this access type, you can set a timeout for waiting for all Load Balancer IPs to be allocated. For details, see Set timeout to get load balancer external IPs and a delay for connecting to the pods .

     The external IP is displayed in the performance test as follows:

   Copy code

   Example of configuring Kubernetes with "NodePortExternalIPs" access type

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="KubernetesAccessType" value="NodePortExternalIPs"/>
         </appSettings>
       </configuration>

3. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

## Set timeout to get load balancer external IPs and a delay for connecting to the pods

When an external traffic policy is set using the "LoadBalancerIngress" access type, you can configure the following settings:

- Add a timeout for getting all Load Balancer IPs. This is the amount of time, in minutes, that LoadRunner Enterprise waits for all Load Balancer IPs to be allocated. This is useful where numerous Docker load generators are provisioned in the test.

- Determine how much time, in seconds, the Controller waits before connecting to the pods. The default delay time is 15 seconds. We recommend increasing the value if you encounter connecting to host (Docker LGs) errors.

To set a timeout for Kubernetes to get load balancer external IPs:

1. Open the kubernetes.config file located in the <LRE server installation path>\dat folder.

2. Under <appsettings>, add a key named "WaitForIngressIPsTimeOut" and enter an integer of 0 or higher. If no value is set, the default timeout is 5 minutes.

3. To set a wait time for connecting to the pods, add a key named "ConnectToPodDelay" and enter a value in seconds.

   Copy code
   Example of timeout for load balancer external IPs and delay connecting to pod:

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="KubernetesAccessType" value="LoadBalancerIngress"/>
           <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
           <add key="ConnectToPodDelay" value="15"/> <!--in seconds-->
         </appSettings>
       </configuration>

4. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

## Add annotations to Kubernetes services

You can use annotations to attach non-identifying metadata to Kubernetes services.

To define annotations in services:

1. Open the kubernetes.config file located in the <LRE server installation path>\dat folder.

2. Under <appsettings>, add a key named "Annotations" and enter a key and value using the following format:

   `<add key="Annotations" value="{'key':'value'}"/>` for a single annotation.

   `<add key="Annotations" value="{'key1':'value1','key2':'value2'}"/>` for multiple annotations.

   Copy code

   Example of defining an annotation:

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="Annotations" value="{'service.beta.kubernetes.io/azure-load-balancer-internal':'true'}" />
          </appSettings>
       </configuration>

3. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

## Disable the service account auto-mount

Available in versions: 2023 R1 and later

By default, a service account is mounted to every pod. If the application doesn’t require access to the Kubernetes API, disable the service account auto-mount in Kubernetes. Disabling service auto-mount hardens cluster accessibility to the API server.

To disable the auto-mount service account token:

1. Open the kubernetes.config file located in the <LRE server installation path>\dat folder.

2. Under <appsettings>, add a key named "AutoMountServiceAccountToken" with a value of "false".

   Copy code
   Example of disabling the service auto-mount:

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="AutomountServiceAccountToken" value="false"/> <!--This is used for hardening cluster accessibility to the API server.-->
           <add key="ConnectToPodDelay" value="15"/> <!--in seconds-->
         </appSettings>
       </configuration>

Back to top

## Kubernetes health checks

Available in versions: 2023 R1 and later

LoadRunner includes the Kubernetes probes which you can use to perform health checks on your Kubernetes application and container. All probes use the TCP socket.

To enable Kubernetes health probes:

1. Open the kubernetes.config file located in the <LRE server installation path>\dat folder.

2. Under <appsettings>, enable the required probes listed below by changing the key value to "true".

   Key | Description
   --- | ---
   "EnableStartupProbe" | The Startup probe determines if the application that runs in a container has started. If it has started, Kubernetes allows other probes to start functioning. Otherwise, the kubelet shuts down and restarts the container. This is used to protect slow starting containers. Default values: initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" failureThreshold="3"
   "EnableLivenessProbe" | The Liveness probe determines if a container is operating. If it does not operate, the kubelet shuts down and restarts the container. Default values: initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" failureThreshold="3"
   "EnableReadinessProbe" | The Readiness probe determines if the application that runs in a container is ready to accept requests. If it is ready, Kubernetes allows matching services to send traffic to the pod. If it is not ready, the endpoints controller removes this pod from all matching services. If a failed pod is not restarted, the performance test fails. Default values: initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" successThreshold="1" failureThreshold="3"

3. To customize probe values, uncomment the selected probes (in the section above <appsettings>), and enter values of your choice.

   Copy code
   Example of startup probe enable with customized failure threshold:

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <!-- Do not modify the configSections -->
         <configSections>
           <section name="StartupProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
           <section name="LivenessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
           <section name="ReadinessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
         </configSections>

         <!-- Uncomment the sections below to apply custom values to probes -->
         <StartupProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" failureThreshold="5" />
         <!-- <LivenessProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" failureThreshold="3" /> -->
         <!-- <ReadinessProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" successThreshold="1" failureThreshold="3" /> -->

         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
           <add key="EnableStartupProbe" value="true"/>
           <add key="EnableLivenessProbe" value="false"/>
           <add key="EnableReadinessProbe" value="false"/>
           <add key="AutomountServiceAccountToken" value="true"/>
         </appSettings>
       </configuration>

Notes and limitations

The following notes and limitations are applicable when using Kubernetes health probes:

Startup probes | Use the Startup probe settings with caution since they can cause permanent failure in the performance test run. In most cases, it should be sufficient to set the "initialDelaySeconds" setting to 60 seconds for the Startup probe. There is no need to use the following key if the Startup probe is used: `<add key=" ConnectToPodDelay" value="5"/> <!--in seconds-->`
Liveness and Readiness probes | These probes should only be used in cases where the cluster policy requires that probes are configured and without them the user cannot run a performance test. These probes do not recover Vusers during a performance test. As a result, we recommend using the Startup probe, and adding about a 60 second delay, to avoid cases where the remote agent service has not started yet. Otherwise the test will fail.
Liveness probe and EKS | Avoid using only the Liveness probe, since this may cause permanent failure in EKS. Example: Copy code Example of kubernetes.config file with only the Liveness probe enabled:
    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <!-- Do not modify the configSections -->
      <configSections>
        <section name="StartupProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
        <section name="LivenessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
        <section name="ReadinessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
      </configSections>

      <!-- Uncomment sections to apply custom values to probes -->
      <!-- <StartupProbe initialDelaySeconds="60" periodSeconds="10" timeoutSeconds="1" failureThreshold="3" /> -->
      <LivenessProbe initialDelaySeconds="120" periodSeconds="10" timeoutSeconds="10" failureThreshold="2" />
      <!-- <ReadinessProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="300" successThreshold="1" failureThreshold="3" /> -->

      <appSettings>
        <add key="DeploymentAPIVersion" value="apps/v1"/>
        <add key="PodAPIVersion" value="v1"/>
        <add key="ServiceAPIVersion" value="v1"/>
        <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
        <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
        <add key="KubernetesAccessType" value="LoadBalancerIngress"/>
        <add key="Annotations" value="{'service.beta.kubernetes.io/aws-load-balancer-internal':'true'}"/>
        <!-- <add key=" ConnectToPodDelay" value="120"/> -->
        <add key="EnableStartupProbe" value="false"/>
        <add key="EnableLivenessProbe" value="true"/>
        <add key="EnableReadinessProbe" value="false"/>
      </appSettings>
    </configuration>

Back to top

## Disable TLS verification

By default, all requests sent to a Kubernetes cluster require a CA certificate and a TLS handshake, otherwise the request from LoadRunner Enterprise to Kubernetes fails. You can disable TLS certificate verification in the Kubernetes configuration file.

Note: For security reasons, we do not recommend disabling TLS verification.

To disable TLS verification for requests sent to a Kubernetes cluster:

1. Open the kubernetes.config file located in the <LRE server installation path>\dat folder.

2. Under <appsettings>, add a key named "EnableSslCertificateVerification" with a value of "false".

   Copy code

   Example of disabling TLS verification:

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="EnableSslCertificateVerification" value="false"/>
          </appSettings>
       </configuration>

3. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

See also:

- Automatic provisioning of Docker hosts

https://admhelp.microfocus.com LoadRunner Enterprise 8942
Tips to improve your search results... LoadRunner Enterprise Help Center LoadRunner Enterprise Help LR Enterprise Help 2023-2023 R1 All View Local Help Center View Online Help Center back Go back in browser history forward Go forward in browser history Remove search highlights next topic Go to next topic previous topic Go to previous topic Print the current topic Version: