Share this page

# Kubernetes configuration settings

This section describes how to configure settings for a Kubernetes image on the OpenText Enterprise Performance Engineering server.

In this topic:

## Configure Kubernetes settings

When using a Kubernetes orchestrator with Dockerized host images, you can configure the following Kubernetes settings in the kubernetes.config file:

- Select the Kubernetes image pull policy.

- Define how Kubernetes containerized load generators are accessed. You can now connect to pods (containerized load generators) through a node port or a load balancer using an external IP.

- Set timeout to get load balancer external IPs and a delay for connecting the Controller to the pods.

- Use annotations to attach non-identifying metadata to Kubernetes services.

- Deactivate the service account auto-mount in Kubernetes.

- Enable Kubernetes health probes.

- Deactivate TLS verification for requests sent to a Kubernetes cluster.

- Specify security settings for all containers in the pod.

Note:

- Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service (EKS) load balancers with External IPs are currently supported. For supported versions, see the Supported Protocols guide available from Support Matrix.

- We recommend making a backup of the kubernetes.config file because the file is deleted during uninstall or upgrade.

- If you upgrade from a version earlier than 25.1, add these lines to the `</configSections>` section of the kubernetes.config file, and then restart IIS (run 'iisreset' in a command line):

  `<section name="GlobalSecurityContext" type="PC.DynamicHostProvision.Common.SecurityContextConfig, PC.DynamicHostProvision.Common"/>`

  `<section name="Tolerations" type="PC.DynamicHostProvision.Common.TolerationsConfig, PC.DynamicHostProvision.Common"/>`

Back to top

## Configure the image pull policy

When creating the pod, you can specify the image pull policy, which guides the kubelet service on how to pull the specified image during an update. Kubernetes has three image pull policies: IfNotPresent, Always, and Never.

To configure the image pull policy:

1. Open the kubernetes.config file located in the <Server_installdir>\dat folder.

2. Under <appsettings>, add a key named ImagePullPolicy with one of the following values.

   AKS | EKS |
   --- | --- | ---
   "Pull If Not Present" | "IfNotPresent" | Kubernetes only pulls the image when it does not already exist on the node. This is the default value if no value is set, or if the pull policy key is missing from the Kubernetes configuration file.
   "Pull Always" | "Always" | Kubernetes always pull the image from the registry when updating the container.
   "Pull Never" | "Never" | Kubernetes never pulls the image.

   Example of configuring the Kubernetes image pull policy for EKS:

   Copy code

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="ImagePullPolicy" value="Always" />
         </appSettings>
       </configuration>

3. If you are using a Kubernetes cluster with a private registry that requires authorization, you also need to perform the following:

   1. In Docker Hub, sign in to your private registry by running `docker login`

   2. Create a secret of your private registry to pull a private image:

      Copy code

          kubectl create secret generic <secret name> \
             --from-file=.dockerconfigjson=<path/to/.docker/config.json> \
             --type=kubernetes.io/dockerconfigjson

   3. Keep a record of your registry secret. You need to add it when configuring the orchestrator settings in Administration (Orchestration > Secret Name box). For details, see Add and configure the orchestrator.

4. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

## Define containerized load generator access

You can define how the Kubernetes pods, or containerized load generators, are accessed based on your cluster configuration. If you use a load balancer such as AKS, you must use the cloud provider's load balancer. If you are using Kubernetes on-premises, you can connect to pods through a node port or a load balancer using an external IP. Node means the machine which runs the pods or containers.

Example: This diagram shows the implementation of the Load balancer service with external IP flow.

To define how containerized load generators are accessed:

1. Open the kubernetes.config file in the <Server_installdir>\dat folder.

2. Under <appsettings>, add a key named "KubernetesAccessType" with one of the following values:

   - "NodePortNodeNames". Exposes the service on an internal IP in the cluster. This is the default value if no value is set, or if the access type key is missing from the Kubernetes configuration file.

   - "NodePortExternalIPs". Exposes services to external clients through a static port on each node's IP.

   - "LoadBalancerIngress". Exposes the service to external clients through the cloud provider's load balancer. This enables provisioning load generators that are set up in different networks to the Controller. When using this access type, you can set a timeout for waiting for all Load Balancer IPs to be allocated. For details, see Set load balancer IP timeout and pod connection delay.

     The external IP is displayed in the performance test as follows:

     Example of configuring Kubernetes with "NodePortExternalIPs" access type:

     Copy code

         <?xml version="1.0" encoding="utf-8" ?>
         <configuration>
           <appSettings>
             <add key="DeploymentAPIVersion" value="apps/v1"/>
             <add key="PodAPIVersion" value="v1"/>
             <add key="ServiceAPIVersion" value="v1"/>
             <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
             <add key="KubernetesAccessType" value="NodePortExternalIPs"/>
           </appSettings>
         </configuration>

3. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

## Set load balancer IP timeout and pod connection delay

When an external traffic policy is set using the "LoadBalancerIngress" access type, you can configure the following settings:

- Add a timeout for getting all Load Balancer IPs. This is the amount of time, in minutes, that OpenText Enterprise Performance Engineering waits for all Load Balancer IPs to be allocated. This is useful where numerous Docker load generators are provisioned in the test.

- Determine how much time, in seconds, the Controller waits before connecting to the pods. The default delay time is 15 seconds. We recommend increasing the value if you encounter connecting to host (Docker LGs) errors.

To set a timeout for Kubernetes to get load balancer external IPs:

1. Open the kubernetes.config file located in the <Server_installdir>\dat folder.

2. Under <appsettings>, add a key named "WaitForIngressIPsTimeOut" and enter an integer of 0 or higher. If no value is set, the default timeout is 5 minutes.

3. To set a wait time for connecting to the pods, add a key named "ConnectToPodDelay" and enter a value in seconds.

   Example of timeout for load balancer external IPs and delay connecting to pod:

   Copy code

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="KubernetesAccessType" value="LoadBalancerIngress"/>
           <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
           <add key="ConnectToPodDelay" value="15"/> <!--in seconds-->
         </appSettings>
       </configuration>

4. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

## Add Kubernetes services annotations

You can use annotations to attach non-identifying metadata to Kubernetes services.

To define annotations in services:

1. Open the kubernetes.config file located in the <Server_installdir>\dat folder.

2. Under <appsettings>, add a key named "Annotations" and enter a key and value using the following format:

   `<add key="Annotations" value="{'key':'value'}"/>` for a single annotation.

   `<add key="Annotations" value="{'key1':'value1','key2':'value2'}"/>` for multiple annotations.

   Example of defining an annotation:

   Copy code

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="Annotations" value="{'service.beta.kubernetes.io/azure-load-balancer-internal':'true'}" />
          </appSettings>
       </configuration>

3. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run 'iisreset' in a command line).

Back to top

## Deactivate the service account auto-mount

By default, a service account is mounted to every pod. If the application doesn’t require access to the Kubernetes API, deactivate the service account auto-mount in Kubernetes. Deactivating service auto-mount hardens cluster accessibility to the API server.

To deactivate the auto-mount service account token:

1. Open the kubernetes.config file located in the <Server_installdir>\dat folder.

2. Under <appsettings>, add a key named "AutoMountServiceAccountToken" with a value of "false".

   Example of deactivating the service auto-mount:

   Copy code

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="AutomountServiceAccountToken" value="false"/> <!--This is used for hardening cluster accessibility to the API server.-->
           <add key="ConnectToPodDelay" value="15"/> <!--in seconds-->
         </appSettings>
       </configuration>

Back to top

## Enable Kubernetes health checks

OpenText Professional Performance Engineering includes the Kubernetes probes which you can use to perform health checks on your Kubernetes application and container. All probes use the TCP socket.

To enable Kubernetes health probes:

1. Open the kubernetes.config file located in the <Server_installdir>\dat folder.

2. Under <appsettings>, enable the required probes listed below by changing the key value to "true".

   Key | Description
   --- | ---
   "EnableStartupProbe" | The Startup probe determines if the application that runs in a container has started. If it has started, Kubernetes allows other probes to start functioning. Otherwise, the kubelet shuts down and restarts the container. This is used to protect slow starting containers. Default values: initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" failureThreshold="3"
   "EnableLivenessProbe" | The Liveness probe determines if a container is operating. If it does not operate, the kubelet shuts down and restarts the container. Default values: initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" failureThreshold="3"
   "EnableReadinessProbe" | The Readiness probe determines if the application that runs in a container is ready to accept requests. If it is ready, Kubernetes allows matching services to send traffic to the pod. If it is not ready, the endpoints controller removes this pod from all matching services. If a failed pod is not restarted, the performance test fails. Default values: initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" successThreshold="1" failureThreshold="3"

3. To customize probe values, uncomment the selected probes (in the section above <appsettings>), and enter values of your choice.

   Example of startup probe enable with customized failure threshold:

   Copy code

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <!-- Do not modify the configSections -->
         <configSections>
           <section name="StartupProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
           <section name="LivenessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
           <section name="ReadinessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
           <section name="GlobalSecurityContext" type="PC.DynamicHostProvision.Common.SecurityContextConfig, PC.DynamicHostProvision.Common"/>
           <section name="Tolerations" type="PC.DynamicHostProvision.Common.TolerationsConfig, PC.DynamicHostProvision.Common"/>
        </configSections>

         <!-- Uncomment the sections below to apply custom values to probes -->
         <StartupProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" failureThreshold="5" />
         <!-- <LivenessProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" failureThreshold="3" /> -->
         <!-- <ReadinessProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="1" successThreshold="1" failureThreshold="3" /> -->

         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
           <add key="EnableStartupProbe" value="true"/>
           <add key="EnableLivenessProbe" value="false"/>
           <add key="EnableReadinessProbe" value="false"/>
           <add key="AutomountServiceAccountToken" value="true"/>
         </appSettings>
       </configuration>

Notes and limitations

The following notes and limitations are applicable when using Kubernetes health probes.

Area | Description
--- | ---
Startup probes | Use the Startup probe settings with caution because they can cause permanent failure in the performance test run. In most cases, it should be sufficient to set the "initialDelaySeconds" setting to 60 seconds for the Startup probe. There is no need to use the following key if the Startup probe is used: `<add key=" ConnectToPodDelay" value="5"/> <!--in seconds-->`
Liveness and Readiness probes | These probes should only be used in cases where the cluster policy requires that probes are configured and without them the user cannot run a performance test. These probes do not recover Vusers during a performance test. As a result, we recommend using the Startup probe, and adding about a 60 second delay, to avoid cases where the remote agent service has not started yet. Otherwise the test will fail.
Liveness probe and EKS | Avoid using only the Liveness probe, because this may cause permanent failure in EKS. Example of kubernetes.config file with only the Liveness probe enabled: Copy code
    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <!-- Do not modify the configSections -->
      <configSections>
        <section name="StartupProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
        <section name="LivenessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
        <section name="ReadinessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
        <section name="GlobalSecurityContext" type="PC.DynamicHostProvision.Common.SecurityContextConfig, PC.DynamicHostProvision.Common"/>
        <section name="Tolerations" type="PC.DynamicHostProvision.Common.TolerationsConfig, PC.DynamicHostProvision.Common"/>
     </configSections>

      <!-- Uncomment sections to apply custom values to probes -->
      <!-- <StartupProbe initialDelaySeconds="60" periodSeconds="10" timeoutSeconds="1" failureThreshold="3" /> -->
      <LivenessProbe initialDelaySeconds="120" periodSeconds="10" timeoutSeconds="10" failureThreshold="2" />
      <!-- <ReadinessProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="300" successThreshold="1" failureThreshold="3" /> -->

      <appSettings>
        <add key="DeploymentAPIVersion" value="apps/v1"/>
        <add key="PodAPIVersion" value="v1"/>
        <add key="ServiceAPIVersion" value="v1"/>
        <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
        <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
        <add key="KubernetesAccessType" value="LoadBalancerIngress"/>
        <add key="Annotations" value="{'service.beta.kubernetes.io/aws-load-balancer-internal':'true'}"/>
        <!-- <add key=" ConnectToPodDelay" value="120"/> -->
        <add key="EnableStartupProbe" value="false"/>
        <add key="EnableLivenessProbe" value="true"/>
        <add key="EnableReadinessProbe" value="false"/>
      </appSettings>
    </configuration>

Back to top

## Deactivate TLS verification

By default, all requests sent to a Kubernetes cluster require a CA certificate and a TLS handshake, otherwise the request from OpenText Enterprise Performance Engineering to Kubernetes fails. You can deactivate TLS certificate verification in the Kubernetes configuration file.

Note: For security reasons, we do not recommend deactivating TLS verification.

To deactivate TLS verification for requests sent to a Kubernetes cluster:

1. Open the kubernetes.config file located in the <Server_installdir>\dat folder.

2. Under <appsettings>, add a key named "EnableSslCertificateVerification" with a value of "false".

   Example of deactivating TLS verification:

   Copy code

       <?xml version="1.0" encoding="utf-8" ?>
       <configuration>
         <appSettings>
           <add key="DeploymentAPIVersion" value="apps/v1"/>
           <add key="PodAPIVersion" value="v1"/>
           <add key="ServiceAPIVersion" value="v1"/>
           <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
           <add key="EnableSslCertificateVerification" value="false"/>
          </appSettings>
       </configuration>

3. Restart IIS after updating the kubernetes.config file so that the changes take effect immediately (run `iisreset` in a command line).

Back to top

## Specify pod container security settings

You can specify the user ID, Group ID, and fsGroup, to provide the security permissions for a specific container or pod.

To define the user permissions for a container or a pod:

1. Open the kubernetes.config file located in the <Server_installdir>\dat folder.

2. Uncomment the GlobalSecurityContext section, and enter a value for the runAsUser field, or use the default user ID ("1000"). All processes in the container will run with the specified user ID.

   Note:

   - This setting is not applicable to Windows OneLG images.

   - If runAsGroup is set, you must also set runAsUser.

3. Restart IIS (run `iisreset` in a command line) for the changes to take effect.

Back to top

## Configure taint and toleration

To set taints and tolerations in Kubernetes (not related to OS node type), follow these steps:

1. Open the kubernetes.config file located in the <Server_installdir>\dat folder.

2. Enable the <Tolerations> section.

3. Specify values for key, operator, and value. If required, include a value for tolerationSeconds.

   Example:

   Copy code

       <Tolerations>
       <Entries>
       <add key="ssd" operator="" value="true" effect="NoSchedule" tolerationSeconds="" />
       <add key="" operator="" value="" effect="" tolerationSeconds="" />
       </Entries>
       </Tolerations>

4. Save the file.

5. Restart IIS (run `iisreset` in a command line) for the changes to take effect.

Note: Make sure that you set a matching taint under your node pool.

For details on running Windows Docker images in your cluster, see Run Windows Docker images.

Back to top

See also:

- Provision Docker hosts automatically

https://admhelp.microfocus.com Enterprise Performance Engineering,LoadRunner Enterprise 8942
Tips to improve your search results...OpenText™ Enterprise Performance Engineering Help Center OpenText™ Enterprise Performance Engineering Help 26.1 All View Local Help Center View Online Help Center Version: