Kubernetes configuration settings

This section describes how to configure settings for a Kubernetes image on the LoadRunner Enterprise server.

Configure Kubernetes settings

When using a Kubernetes orchestrator with Dockerized host images, you can configure the following Kubernetes settings in the kubernetes.config file:

  • Select the Kubernetes image pull policy.

  • Define how Kubernetes containerized load generators are accessed. You can now connect to pods (containerized load generators) through a node port or a load balancer using an external IP.

  • Set timeout to get load balancer external IPs and a delay for connecting the Controller to the pods.

  • Use annotations to attach non-identifying metadata to Kubernetes services.

  • Deactivate the service account auto-mount in Kubernetes.

  • Enable Kubernetes health probes.

  • Deactivate TLS verification for requests sent to a Kubernetes cluster.

  • Specify security settings for all containers in the pod.

Note:  

  • Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service (EKS) load balancers with External IPs are currently supported. For supported versions, see the Supported Protocols guide available from the Support Matrix.

  • We recommend making a backup of the kubernetes.config file because the file is deleted during uninstall or upgrade.

Back to top

Configure the image pull policy

When creating the pod, you can specify the image pull policy, which guides the kubelet service on how to pull the specified image during an update. Kubernetes has three image pull policies: IfNotPresent, Always, and Never.

To configure the image pull policy:

  1. Open the kubernetes.config file located in the <LRE_server_installdir>\dat folder.

  2. Under <appsettings>, add a key named "ImagePullPolicy" with one of the following values:

    AKS EKS  

    "Pull If Not Present"

    "IfNotPresent"

    Kubernetes only pulls the image when it does not already exist on the node. This is the default value if no value is set, or if the pull policy key is missing from the Kubernetes configuration file.

    "Pull Always"

    "Always"

    Kubernetes always pull the image from the registry when updating the container.

    "Pull Never"

    "Never" Kubernetes never pulls the image.
    Copy code
    Example of configuring the Kubernetes image pull policy for EKS:
    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <appSettings>
        <add key="DeploymentAPIVersion" value="apps/v1"/>
        <add key="PodAPIVersion" value="v1"/>
        <add key="ServiceAPIVersion" value="v1"/>
        <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
        <add key="ImagePullPolicy" value="Always" />
      </appSettings>
    </configuration>
  3. If you are using a Kubernetes cluster with a private registry that requires authorization, you also need to perform the following:

    1. In Docker Hub, log in to your private registry by running docker login

    2. Create a secret of your private registry to pull a private image:

      Copy code
      kubectl create secret generic <secret name> \   
         --from-file=.dockerconfigjson=<path/to/.docker/config.json> \   
         --type=kubernetes.io/dockerconfigjson
    3. Keep a record of your registry secret. You need to add it when configuring the orchestrator settings in LoadRunner Enterprise Administration (Orchestration > Secret Name box). For details, see Add and configure the orchestrator.

  4. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

Define containerized load generator access

You can define how the Kubernetes pods, or containerized load generators, are accessed based on your cluster configuration. If you use a load balancer such as AKS, you must use the cloud provider's load balancer. If you are using Kubernetes on-premises, you can connect to pods through a node port or a load balancer using an external IP. Node means the machine which runs the pods or containers.

Example: This diagram shows the implementation of the Load balancer service with external IP flow:

To define how containerized load generators are accessed:

  1. Open the kubernetes.config file in the <LRE_server_installdir>\dat folder.

  2. Under <appsettings>, add a key named "KubernetesAccessType" with one of the following values:

    • "NodePortNodeNames". Exposes the service on an internal IP in the cluster. This is the default value if no value is set, or if the access type key is missing from the Kubernetes configuration file.

    • "NodePortExternalIPs". Exposes services to external clients through a static port on each node's IP.

    • "LoadBalancerIngress". Exposes the service to external clients through the cloud provider's load balancer. This enables provisioning load generators that are set up in different networks to the Controller. When using this access type, you can set a timeout for waiting for all Load Balancer IPs to be allocated. For details, see Set load balancer IP timeout and pod connection delay.

      The external IP is displayed in the performance test as follows:

    Copy code

    Example of configuring Kubernetes with "NodePortExternalIPs" access type

    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <appSettings>
        <add key="DeploymentAPIVersion" value="apps/v1"/>
        <add key="PodAPIVersion" value="v1"/>
        <add key="ServiceAPIVersion" value="v1"/>
        <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
        <add key="KubernetesAccessType" value="NodePortExternalIPs"/>
      </appSettings>
    </configuration>
  3. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

Set load balancer IP timeout and pod connection delay

When an external traffic policy is set using the "LoadBalancerIngress" access type, you can configure the following settings:

  • Add a timeout for getting all Load Balancer IPs. This is the amount of time, in minutes, that LoadRunner Enterprise waits for all Load Balancer IPs to be allocated. This is useful where numerous Docker load generators are provisioned in the test.

  • Determine how much time, in seconds, the Controller waits before connecting to the pods. The default delay time is 15 seconds. We recommend increasing the value if you encounter connecting to host (Docker LGs) errors.

To set a timeout for Kubernetes to get load balancer external IPs:

  1. Open the kubernetes.config file located in the <LRE_server_installdir>\dat folder.

  2. Under <appsettings>, add a key named "WaitForIngressIPsTimeOut" and enter an integer of 0 or higher. If no value is set, the default timeout is 5 minutes.

  3. To set a wait time for connecting to the pods, add a key named "ConnectToPodDelay" and enter a value in seconds.

    Copy code
    Example of timeout for load balancer external IPs and delay connecting to pod:
    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <appSettings>
        <add key="DeploymentAPIVersion" value="apps/v1"/>
        <add key="PodAPIVersion" value="v1"/>
        <add key="ServiceAPIVersion" value="v1"/>
        <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
        <add key="KubernetesAccessType" value="LoadBalancerIngress"/>
        <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
        <add key="ConnectToPodDelay" value="15"/> <!--in seconds-->
      </appSettings>
    </configuration>
  4. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

Add Kubernetes services annotations

You can use annotations to attach non-identifying metadata to Kubernetes services.

To define annotations in services:

  1. Open the kubernetes.config file located in the <LRE_server_installdir>\dat folder.

  2. Under <appsettings>, add a key named "Annotations" and enter a key and value using the following format:

    <add key="Annotations" value="{'key':'value'}"/> for a single annotation.

    <add key="Annotations" value="{'key1':'value1','key2':'value2'}"/> for multiple annotations.

    Copy code

    Example of defining an annotation:

    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <appSettings>
        <add key="DeploymentAPIVersion" value="apps/v1"/>
        <add key="PodAPIVersion" value="v1"/>
        <add key="ServiceAPIVersion" value="v1"/>
        <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
        <add key="Annotations" value="{'service.beta.kubernetes.io/azure-load-balancer-internal':'true'}" />
       </appSettings>
    </configuration>
  3. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

Deactivate the service account auto-mount

By default, a service account is mounted to every pod. If the application doesn’t require access to the Kubernetes API, deactivate the service account auto-mount in Kubernetes. Deactivating service auto-mount hardens cluster accessibility to the API server.

To deactivate the auto-mount service account token:

  1. Open the kubernetes.config file located in the <LRE_server_installdir>\dat folder.

  2. Under <appsettings>, add a key named "AutoMountServiceAccountToken" with a value of "false".

    Copy code
    Example of deactivating the service auto-mount:
    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <appSettings>
        <add key="DeploymentAPIVersion" value="apps/v1"/>
        <add key="PodAPIVersion" value="v1"/>
        <add key="ServiceAPIVersion" value="v1"/>
        <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
        <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
        <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
        <add key="AutomountServiceAccountToken" value="false"/> <!--This is used for hardening cluster accessibility to the API server.-->
        <add key="ConnectToPodDelay" value="15"/> <!--in seconds-->
      </appSettings>
    </configuration>

Back to top

Enable Kubernetes health checks

LoadRunner includes the Kubernetes probes which you can use to perform health checks on your Kubernetes application and container. All probes use the TCP socket.

To enable Kubernetes health probes:

  1. Open the kubernetes.config file located in the <LRE_server_installdir>\dat folder.

  2. Under <appsettings>, enable the required probes listed below by changing the key value to "true".

    Key Description
    "EnableStartupProbe"

    The Startup probe determines if the application that runs in a container has started. If it has started, Kubernetes allows other probes to start functioning. Otherwise, the kubelet shuts down and restarts the container. This is used to protect slow starting containers.

    Default values:

    • initialDelaySeconds="0"

    • periodSeconds="10"

    • timeoutSeconds="1"

    • failureThreshold="3"

    "EnableLivenessProbe"

    The Liveness probe determines if a container is operating. If it does not operate, the kubelet shuts down and restarts the container.

    Default values:

    • initialDelaySeconds="0"

    • periodSeconds="10"

    • timeoutSeconds="1"

    • failureThreshold="3"

    "EnableReadinessProbe"

    The Readiness probe determines if the application that runs in a container is ready to accept requests. If it is ready, Kubernetes allows matching services to send traffic to the pod. If it is not ready, the endpoints controller removes this pod from all matching services. If a failed pod is not restarted, the performance test fails.

    Default values:

    • initialDelaySeconds="0"

    • periodSeconds="10"

    • timeoutSeconds="1"

    • successThreshold="1"

    • failureThreshold="3"

  3. To customize probe values, uncomment the selected probes (in the section above <appsettings>), and enter values of your choice.

    Copy code
    Example of startup probe enable with customized failure threshold:
    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <!-- Do not modify the configSections -->
      <configSections>
        <section name="StartupProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
        <section name="LivenessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
        <section name="ReadinessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
      </configSections>

      <!-- Uncomment the sections below to apply custom values to probes -->
      <StartupProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" failureThreshold="5" />
      <!-- <LivenessProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" failureThreshold="3" /> -->
      <!-- <ReadinessProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" successThreshold="1" failureThreshold="3" /> -->

      <appSettings>
        <add key="DeploymentAPIVersion" value="apps/v1"/>
        <add key="PodAPIVersion" value="v1"/>
        <add key="ServiceAPIVersion" value="v1"/>
        <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
        <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
        <add key="EnableStartupProbe" value="true"/>
        <add key="EnableLivenessProbe" value="false"/>
        <add key="EnableReadinessProbe" value="false"/>
        <add key="AutomountServiceAccountToken" value="true"/>
      </appSettings>
    </configuration>

Notes and limitations

The following notes and limitations are applicable when using Kubernetes health probes:

Startup probes

Use the Startup probe settings with caution because they can cause permanent failure in the performance test run. In most cases, it should be sufficient to set the "initialDelaySeconds" setting to 60 seconds for the Startup probe. There is no need to use the following key if the Startup probe is used:

<add key=" ConnectToPodDelay" value="5"/> <!--in seconds-->

Liveness and Readiness probes
  • These probes should only be used in cases where the cluster policy requires that probes are configured and without them the user cannot run a performance test.

  • These probes do not recover Vusers during a performance test. As a result, we recommend using the Startup probe, and adding about a 60 second delay, to avoid cases where the remote agent service has not started yet. Otherwise the test will fail.

Liveness probe and EKS

Avoid using only the Liveness probe, because this may cause permanent failure in EKS.

Back to top

Deactivate TLS verification

By default, all requests sent to a Kubernetes cluster require a CA certificate and a TLS handshake, otherwise the request from LoadRunner Enterprise to Kubernetes fails. You can deactivate TLS certificate verification in the Kubernetes configuration file.

Note: For security reasons, we do not recommend deactivating TLS verification.

To deactivate TLS verification for requests sent to a Kubernetes cluster:

  1. Open the kubernetes.config file located in the <LRE_server_installdir>\dat folder.

  2. Under <appsettings>, add a key named "EnableSslCertificateVerification" with a value of "false".

    Copy code

    Example of deactivating TLS verification:

    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <appSettings>
        <add key="DeploymentAPIVersion" value="apps/v1"/>
        <add key="PodAPIVersion" value="v1"/>
        <add key="ServiceAPIVersion" value="v1"/>
        <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
        <add key="EnableSslCertificateVerification" value="false"/>
       </appSettings>
    </configuration>
  3. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

Specify pod container security settings

You can specify the user ID to provide the security permissions for a specific container or pod.

To define the user permissions for a container or a pod:

  1. Open the kubernetes.config file located in the <LRE_server_installdir>\dat folder.

  2. Uncomment the GlobalSecurityContext section, and enter a value for the runAsUser field, or use the default user ID ("1000"). All processes in the container will run with the specified user ID.

    Note:  

    • This setting is not applicable to Windows OneLG images.

    • Changing the runAsUser value to "-1" or any other negative value negates the setting (it is the same as commenting out runAsUser).

  3. Restart IIS (run 'iisreset' in a command line) for the changes take effect.

Back to top

See also: