Kubernetes configuration settings

This section describes how to configure settings for a Kubernetes image on the OpenText Enterprise Performance Engineering server.

Configure Kubernetes settings

When using a Kubernetes orchestrator with Dockerized host images, you configure the following Kubernetes behavior in the kubernetes.config file. The configuration supports the following capabilities:

• Image pull policy configuration

• Access to containerized load generators

• Load balancer IP allocation timeout and connection delays

• Kubernetes service annotations

• Service account auto‑mount control

• Kubernetes health probes

• TLS verification control

• Pod and container security context settings

• Persistent storage using PersistentVolumeClaims (PVCs)

• Node taints and tolerations

Configure the image pull policy

The image pull policy determines how Kubernetes pulls container images during pod creation or updates.

To configure the image pull policy:

1. Open kubernetes.config in the <Server_installdir>\dat folder.

2. Add the ImagePullPolicy key under <appsettings>.

   Example - EKS:

   Copy code

   <?xml version="1.0" encoding="utf-8" ?>
   <configuration>
     <appSettings>
       <add key="DeploymentAPIVersion" value="apps/v1"/>
       <add key="PodAPIVersion" value="v1"/>
       <add key="ServiceAPIVersion" value="v1"/>
       <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
       <add key="ImagePullPolicy" value="Always" />
     </appSettings>
   </configuration>

3. If your cluster uses a private registry:

   1. In Docker Hub, sign in to your private registry by running docker login.

   2. Create a Kubernetes registry secret:

      Copy code

      kubectl create secret generic <secret name> \   
         --from-file=.dockerconfigjson=<path/to/.docker/config.json> \   
         --type=kubernetes.io/dockerconfigjson

   3. Record the secret name and configure it in Administration > Orchestration. For details, see Add and configure the orchestrator.

4. Restart IIS after updating kubernetes.config for changes to take effect. Run iisreset in a command line.

Define access to containerized load generators

You can define how Kubernetes pods, or containerized load generators, are accessed based on your cluster configuration. If you use a load balancer such as AKS, use the cloud provider load balancer. For on-premises Kubernetes, you can connect to pods through a node port or a load balancer with an external IP. A node is the machine that runs the pods or containers.

Example: This diagram shows the implementation of the load balancer service with external IP flow.

To define how containerized load generators are accessed:

1. Open kubernetes.config in the <Server_installdir>\dat folder.

2. Under <appsettings>, add the KubernetesAccessType key with one of the following values.

   Value	Description
   NodePortNodeNames	Exposes services on an internal cluster IP. This is the default if no value is set, or if the access type key is missing.
   NodePortExternalIPs	Exposes services through a static port on each node’s external IP.
   LoadBalancerIngress	Uses the cloud provider’s load balancer. This enables provisioning load generators that are set up in different networks to the Controller. Note: Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service (EKS) load balancers with External IPs are currently supported. For supported versions, see the Supported Protocols guide available from Support Matrix. When using this access type, you can set a timeout for waiting for all Load Balancer IPs to be allocated. For details, see Set load balancer IP timeout and pod connection delay.

   The external IP is displayed in the performance test as follows:

   

   Copy code

   Example of configuring Kubernetes with the NodePortExternalIPs access type.

   <?xml version="1.0" encoding="utf-8" ?>
   <configuration>
     <appSettings>
       <add key="DeploymentAPIVersion" value="apps/v1"/>
       <add key="PodAPIVersion" value="v1"/>
       <add key="ServiceAPIVersion" value="v1"/>
       <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
       <add key="KubernetesAccessType" value="NodePortExternalIPs"/>
     </appSettings>
   </configuration>

3. Restart IIS after updating kubernetes.config for changes to take effect. Run iisreset in a command line.

Set load balancer IP timeout and pod connection delay

When using LoadBalancerIngress, you can configure the load balancer IP timeout and pod connection delay.

To set a timeout for Kubernetes to get load balancer external IPs:

1. Open kubernetes.config in the <Server_installdir>\dat folder.

2. Under <appsettings>, add one of the following keys.

   Key	Description
   WaitForIngressIPsTimeOut	Timeout, in minutes, for allocating all load balancer IPs (default: 5). This is useful where numerous Docker load generators are provisioned in the test.
   ConnectToPodDelay	Delay, in seconds, before the Controller connects to pods (default: 15). We recommend increasing the value if you encounter connecting to host (Docker LGs) errors.

   Copy code

   Example of timeout for load balancer external IPs and delay connecting to pod:

   <?xml version="1.0" encoding="utf-8" ?>
   <configuration>
     <appSettings>
       <add key="DeploymentAPIVersion" value="apps/v1"/>
       <add key="PodAPIVersion" value="v1"/>
       <add key="ServiceAPIVersion" value="v1"/>
       <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
       <add key="KubernetesAccessType" value="LoadBalancerIngress"/>
       <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
       <add key="ConnectToPodDelay" value="15"/> <!--in seconds-->
     </appSettings>
   </configuration>

3. Restart IIS after updating kubernetes.config for changes to take effect. Run iisreset in a command line.

Add Kubernetes service annotations

You can use annotations to attach non-identifying metadata to Kubernetes services.

To define annotations in services:

1. Open kubernetes.config in the <Server_installdir>\dat folder.

2. Under <appsettings>, add a key named Annotations and enter a key and value using the following format.

   Copy code

   For a single annotation:

   <add key="Annotations" value="{'key':'value'}"/>

   Copy code

   For multiple annotations:

   <add key="Annotations" value="{'key1':'value1','key2':'value2'}"/>

   Copy code

   Example of Kubernetes service annotations:

   <?xml version="1.0" encoding="utf-8" ?>
   <configuration>
     <appSettings>
       <add key="DeploymentAPIVersion" value="apps/v1"/>
       <add key="PodAPIVersion" value="v1"/>
       <add key="ServiceAPIVersion" value="v1"/>
       <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
       <add key="Annotations" value="{'service.beta.kubernetes.io/azure-load-balancer-internal':'true'}" />
      </appSettings>
   </configuration>

3. Restart IIS after updating kubernetes.config for changes to take effect. Run iisreset in a command line.

Deactivate service account auto-mount

By default, Kubernetes mounts a service account token into every pod. If API access is not required, deactivate auto‑mount for improved security. Deactivating service auto-mount hardens cluster accessibility to the API server.

To deactivate the auto-mount service account token:

1. Open kubernetes.config in the <Server_installdir>\dat folder.

2. Under <appsettings>, add a key named AutoMountServiceAccountToken with a value of false.

   Copy code

   Example of deactivating the service auto-mount:

   <?xml version="1.0" encoding="utf-8" ?>
   <configuration>
     <appSettings>
       <add key="DeploymentAPIVersion" value="apps/v1"/>
       <add key="PodAPIVersion" value="v1"/>
       <add key="ServiceAPIVersion" value="v1"/>
       <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
       <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
       <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
       <add key="AutomountServiceAccountToken" value="false"/> <!--This is used for hardening cluster accessibility to the API server.-->
       <add key="ConnectToPodDelay" value="15"/> <!--in seconds-->
     </appSettings>
   </configuration>

3. Restart IIS after updating kubernetes.config for changes to take effect. Run iisreset in a command line.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <!-- Do not modify the configSections -->
  <configSections>
    <section name="StartupProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
    <section name="LivenessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
    <section name="ReadinessProbe" type="PC.DynamicHostProvision.Common.ProbeConfig, PC.DynamicHostProvision.Common"/>
    <section name="GlobalSecurityContext" type="PC.DynamicHostProvision.Common.SecurityContextConfig, PC.DynamicHostProvision.Common"/>
    <section name="Tolerations" type="PC.DynamicHostProvision.Common.TolerationsConfig, PC.DynamicHostProvision.Common"/>
 </configSections>

  <!-- Uncomment sections to apply custom values to probes -->
  <!-- <StartupProbe initialDelaySeconds="60" periodSeconds="10" timeoutSeconds="1" failureThreshold="3" /> -->
  <LivenessProbe initialDelaySeconds="120" periodSeconds="10" timeoutSeconds="10" failureThreshold="2" />
  <!-- <ReadinessProbe initialDelaySeconds="0" periodSeconds="10" timeoutSeconds="300" successThreshold="1" failureThreshold="3" /> -->

  <appSettings>
    <add key="DeploymentAPIVersion" value="apps/v1"/>
    <add key="PodAPIVersion" value="v1"/>
    <add key="ServiceAPIVersion" value="v1"/>
    <add key="CreateDeploymentTimeout" value="10"/> <!--in seconds-->
    <add key="WaitForIngressIPsTimeOut" value="5"/> <!--in minutes-->
    <add key="KubernetesAccessType" value="LoadBalancerIngress"/>
    <add key="Annotations" value="{'service.beta.kubernetes.io/aws-load-balancer-internal':'true'}"/>
    <!-- <add key=" ConnectToPodDelay" value="120"/> --> 
    <add key="EnableStartupProbe" value="false"/>
    <add key="EnableLivenessProbe" value="true"/>
    <add key="EnableReadinessProbe" value="false"/>
  </appSettings>
</configuration>

<GlobalSecurityContext>
    runAsUser="1001"
    runAsGroup="1001"
    runAsNonRoot="true"
    fsGroup=""
    allowPrivilegeEscalation="false"
    privileged="false">
    <seccompProfile type="RuntimeDefault" localhostProfile="" />
    <capabilities add="" drop="['ALL']" />
</GlobalSecurityContext>

You control which PVC is mounted, where it is mounted (mountPath), and whether the pod can write to it (readOnly).

The integration flow is: Pod -> PersistentVolumeClaim -> PersistentVolume -> Storage backend.

To enable persistent volumes in Kubernetes integration:

1. Open kubernetes.config in the <Server_installdir>\dat folder.

2. Uncomment the <PersistentVolumes> section.

   Copy code

   <PersistentVolumes>
     <Entries>
       <add name="<logical name>"
            claimName="<PersistentVolumeClaim name>"
            mountPath="<path inside the container>"
            readOnly="true|false" />
     </Entries>
   </PersistentVolumes>

3. Set values for the following attributes.

   Attribute	Description
   name	Enter any logical name. A name that references the PVC helps identify the mount.
   claimName	Enter your PVC name.
   mountPath	Enter the path inside the container, for example /mnt/nas.
   readOnly	Optional. Set to true or false.

4. Restart IIS after updating kubernetes.config for changes to take effect. Run iisreset in a command line.

For details on running Windows Docker images in your cluster, see Run Windows Docker images.

Limitations

If the following error occurs:

container has runAsNonRoot and image has non-numeric user (lr_user);
cannot verify user is non-root

Cause:

The container image uses a named user instead of a numeric UID. When runAsNonRoot=true is set, Kubernetes cannot verify that the user is not root.

Resolution:

• Use an image with a numeric user ID, or

• Set runAsNonRoot="false" in GlobalSecurityContext.

See also

• Provision Docker hosts automatically