HTTP(S) Proxy Connector Configuration

The HTTP proxy connector is very useful for the virtualization of REST and SOAP services when the client can be configured to access the services through a proxy.

Modes of operation

The HTTP proxy connector can be used in two different ways: The usual non-transparent proxy, when the proxy server host and port are provided to the client, and a transparent proxy, where the traffic is routed through the proxy server at the TCP layer (L3) without the HTTP client knowing that its connections are being redirected.

Non-transparent HTTP(S) proxy

The most common way to redirect HTTP(S) traffic through a proxy server is to provide the client with directions to use the proxy when sending HTTP(S) requests. The instructions depend on the client platform and implementation.

The following table summarizes the way the proxy can be configured based on the client platform:

Client	Proxy Configuration
Java application	Java properties: http.proxyHost, http.proxyPort, http.nonProxyHosts, https.proxyHost, https.proxyPort. See Java documentation.
NodeJS application	Use one of techniques described in this topic.
Browser application	Set system proxy (Chrome, Edge) or the browser proxy configuration (Firefox).

Transparent HTTP(S) proxy

The transparent HTTP(S) proxy works by redirecting all requests from the computer running the client code through the proxy server at the TCP/IP level. The application itself does not have to know about the proxy server at all. It opens a connection to the real server, but it is redirected to the transparent proxy at the system level.

A good example is a Linux Docker host machine. Traffic to ports 80 (HTTP) and 443 (HTTPS) can be redirected to the proxy connector listening at 192.168.1.107 with the following iptables commands:

iptables -t nat -A OUTPUT -p tcp --dport 80 -j DNAT --to-destination 192.168.1.107:9010 
iptables -t nat -A OUTPUT -p tcp --dport 443 -j DNAT --to-destination 192.168.1.107:9011 

In this example, requests from the Home Assistant application to the https://owner-api.teslamotors.com/ secure endpoint will be redirected by iptables through the secure transparent proxy at 192.168.1.107:9011 and requests to the http://192.168.1.150/dyn plain HTTP endpoint will go through the unsecure proxy at 192.168.1.107:9010.

If the Docker application consists of multiple containers communicating with each other, it is recommended to exclude this traffic from the redirection as the internal Docker network is not visible from outside and the proxy connector will not be able to redirect the request back to the internal addresses. When the internal Docker network is using addresses from the 172.0.0.0/8 space, the above commands would then be as follows:

iptables -t nat -A OUTPUT -p tcp ! -d 172.0.0.0/8 --dport 80 -j DNAT --to-destination 192.168.1.107:9010 
iptables -t nat -A OUTPUT -p tcp ! -d 172.0.0.0/8 --dport 443 -j DNAT --to-destination 192.168.1.107:9011 

You can list current iptables rules with:

iptables -nvL -t nat

For more information and options see the iptables documentation.

The proxy connector works both as a non-transparent proxy and a transparent proxy for plain HTTP traffic at the port configured with the bindPort property. For a transparent HTTPS (secure) proxy, a separate secure port must be set up with the bindSecurePort configuration property. For example:

"connector": [
  {
    "id": "proxyConnector",
    "connectorType": "httpProxy",
    "properties": {
      "bindPort": 9010,
      "bindSecurePort": 9011,
      "keyStore": "...",
      "keyStorePassword": "...",
      "privateKeyPassword": "..."
    }
  }
]

Note

The client must be forced to ignore SSL errors, as the transparent HTTPS proxy presents certificates that are invalid from the client's point of view. The certificate belongs to the "sv.lab" host rather than to the server hosting the real service. This is true for every request going through the transparent HTTPS proxy, both to virtual services as well as to the endpoints that were not virtualized.

Secure (HTTPS) traffic handling

While capturing plain HTTP traffic is straightforward, most of the service endpoints nowadays use encrypted HTTPS communications. The proxy connector supports virtual services with secure endpoints by decrypting the encrypted traffic:

Non-transparent Proxy

• All requests to servers hosting at least one virtual service get hijacked and decrypted, regardless of the service mode.
• The proxy connector creates a fake certificate for the host, signed by a certification authority configured by the keyStore property (for details, see below). The client must be set up to trust the certification authority, or to ignore SSL errors.
• Requests to servers that do not host any virtual service are not decrypted. This limits the endpoint discovery functionality. The proxy connector does not see full service URLs, so only the https://host:port part of the URL is reported in service discovery.
  • Tip: To discover full endpoint URLs on a secure server, create a bogus virtual service at that server (https://host:port/fooService). The connector starts decrypting all traffic to host:port and will be able to discover and display complete URLs of all services at that server. For example, enable discovery while learning (a non-existent) https://www.google.com/fooService to see all secure endpoints accessed at the www.google.com host:

    sv-capture -r https://www.google.com/fooService -d

• With a non-transparent HTTPS proxy, the initial communication with the proxy is always unsecure. The connection gets upgraded using the HTTP CONNECT method to establish a secure connection with the server.

Transparent Proxy

• All HTTPS requests going through the transparent proxy at the port specified by the bindSecurePort configuration property get decrypted.
• The proxy connector presents itself with a certificate belonging to the sv.lab host, which never matches the requested endpoint. The client must be set-up to ignore SSL errors.

In order to be able to capture and simulate secure HTTPS traffic, the proxy connector must be configured with the following properties:

• keyStore: (byte[]) serialized Java keystore with private key for fake SSL certificate generation
• keyStorePassword: (string) password for Java keystore
• privateKeyPassword: (string) password for private key stored in keystore

For example:

"connector": [
  {
    "id": "proxyConnector",
    "connectorType": "httpProxy",
    "properties": {
      "bindPort": 9010,
      "keyStore": "/u3+7QAAAAIAAAABAAAAAQABMQAAAWGJNngwAAAJhjCCCYIwDgYKKwYB...",
      "keyStorePassword": "sv-lab-changeit",
      "privateKeyPassword": "sv-lab-changeit"
    }
  }
]

Note

When you use the sv-capture tool for the discovery or learning of an HTTPS endpoint, it creates an sv-lab.json configuration file with a keystore with the 'sv-lab-changeit' keystore/private key passwords. The keystore is available in the bin/sv-capture-keystore.jks file next to the sv-capture tool and the exported public key is in the bin/sv-capture.pem file.

The method of configuring the client application to trust the fake certificates generated by the proxy connector depends on the platform used. Basically, the CA certificate must be imported to a client's truststore. The following table describes how this can be done on different platforms:

Client	Trust Store Configuration
Java application	Use the -Djavax.net.ssl.trustStore=path_to/sv-capture-keystore.jks Java property.
NodeJS application	Set the sv-capture.pem path to the NODE_EXTRA_CA_CERTS system environment variable. See the NodeJS documentation.
Browser application	Import the sv-capture.pem certificate as a trusted root CA on your system.

Caution

Establishing trust to the provided sv-capture.pem certificate (or the sv-capture-keystore.jks keystore, respectively) opens your system to a variety of attacks by anyone possessing the private key. It is good practice to generate and use your own keystore in your test environments.

Using a custom CA certificate

The sv-capture tool uses the bin/sv-capture-tool.jks keystore with a root certification authority certificate/private key by default. In includes the Base64-encoded keystore in the generated sv-lab.json configuration files.

It is good practice to generate your own .jks and .pem files and replace the default values in your sv-lab.json configuration files.

You can generate a new keystore using the Java keytool:

keytool -genkey -alias sv-lab -keyalg RSA -keystore my-keystore.jks -keysize 4096 -validity 36500

You can export the CA certificate for clients in .pem format:

keytool -exportcert -keystore my-keystore.jks -alias sv-lab -rfc -file my-certificate.pem```

Keystore/truststore format

The value of the keyStore, clientKeyStore, and trustStore properties is a Base64-encoded Java keystore. You can use the online converter. In Linux, use the following command:

base64 -e sv-capture-keystore.jks output.txt

In Windows, you can use the certutil command, but you must strip out the ---BEGIN CERTIFICATE--- and ---END CERTIFICATE--- text from output:

certutil -encode sv-capture-keystore.jks output.txt

Upstream proxy

When the test environment with virtual lab is behind a proxy (for example, a corporate firewall), the proxy connector must be configured with the following properties to be able to reach the real services through that proxy:

• proxyHost: (string) upstream proxy host
• proxyPort: (int) upstream proxy port

Client authentication at real endpoint

The real service may require client authentication. For certain types of authentication, the proxy connector must be correctly configured:

Client Authentication	Connector Configuration
HTTP Basic	Use the username and password connector configuration properties.
Mutual SSL Authentication / mTLS	Use the clientKeyStore, clientKeyStorePassword and clientPrivateKeyPassword configuration properties. The keystore format is the same as for the keyStore property described above.
OAuth	No connector configuration is required, the authentication token is passed in the HTTP header.

Service authenticity validation

In the test environment, it is usually not necessary to verify true authenticity of the real service. However, you can provide the trustStore and trustStorePassword properties to the connector to enable authentication of endpoints. Connections to endpoints that can not present a certificate trusted by the truststore provided will not be established. The format of the truststore is the same as the keystore format for the keyStore property described above.

Service discovery

The HTTP proxy connector enables discovery of which service endpoints are being accessed by the client. The discovery mode is usually used with the sv-capture tool. It can also be enabled with the correct connector configuration. The discovered endpoints are recorded and displayed in a run log:

• Setting the endpointDiscovery=true property enables discovery of endpoints for which there is no virtual service created in the virtual lab. The output is filtered so that each endpoint is displayed only on first access.
• Setting the verboseDiscovery=true enables the more verbose discover mode that records all access to non-virtualized endpoints, not just the ones "yet unseen".

Note

The discovery of secure endpoints has the following limitations:

• When the discovered endpoint is at a host that does not host any virtual service in the virtual lab, the secure connection to that host is not decrypted by the HTTP connector and therefore only the host is shown rather than the full endpoint URL.
• When the discovered endpoint is at a host that hosts some virtual services, the full endpoint URL is shown in the output.

HTTP proxy connector configuration properties

Mandatory properties are bold. Other properties are optional.

Property	Default Value	Description
bindPort		(int) The port on which a connector is listening for incoming requests for non-transparent HTTP and HTTPS proxy and transparent (plain) HTTP proxy
bindSecurePort		(int) Optional additional port on which the connector can listen for incoming requests using SSL to work as a transparent HTTPS proxy
keyStore		(byte[]) Serialized Java keystore with private key for fake SSL certificate generation
keyStorePassword		(string) Password for Java keystore
privateKeyPassword		(string) Password for a private key stored in keystore
proxyHost		(string) Upstream proxy host to use when connecting a real service
proxyPort		(int) Upstream proxy port to use when connecting a real service
username		(string) Client's username for authentication at a real service
password		(string) Client's password for authentication at a real service
clientKeyStore		(byte[]) Serialized Java keystore with client's private key for SSL connection to a real service
clientKeyStorePassword		(string) Password for client's keystore
clientPrivateKeyPassword		(string) Password for private key stored in client's keystore
trustStore		(byte[]) Serialized Java truststore with private key for validation of a real service certificate
trustStorePassword		(string) Password for Java truststore
responseTimeout	50 000	(int) How many milliseconds the connector waits fora response before closing the connection
connectionTimeout	5 000	(int) How many milliseconds the connector attempts to connect to the remote party when initiating a connection
readTimeout	5 000	(int) Timeout when reading from the network, in milliseconds
writeTimeout	5 000	(int) Timeout when writing to the network, in milliseconds
endpointDiscovery	false	(boolean) Endpoint discovery - record all unbound endpoints in the run log
verboseDiscovery	false	(boolean) Endpoint discovery should be more verbose - typically record all endpoints and not just the "yet unseen" ones