REST Protocol Discovery And Learning

Watch a video explaining motivation & basic steps of recording a virtual service in SV Lab:

This topic describes how to create a REST virtual service through learning.

Overview and prerequisites

To create a REST virtual service through learning, use the SV capture tool, a sv-capture script located in the Lab/bin directory. It is a light-weight client of the SV Lab server designed for recording and learning application scenarios. It also lets you run a simulation of recorded scenarios in order to test their functionality.

There are two ways to perform REST protocol virtualization, depending on the connector type: HTTP proxy (default) or HTTP gateway. For a comparison, see HTTP_PROXY vs. HTTP_GATEWAY.

Prerequisites:

Make sure you have the SV Lab server up and running. For details, see Lab server reference.
If you need to record traffic on HTTPS connections, we recommend that you add the SV Capture Utility certificate among the Trusted Root Certification Authorities.
By default, the SV capture utility expects the SV Lab server API endpoint to listen at https://localhost:8445. If necessary, modify the endpoint address by adding the -u <SV Lab Server URL> parameter to your command.

To inquire about an unsupported protocol, contact the Service Virtualization Development team.

Learning using a proxy

This is the default and preferred method for learning and simulation of REST services. With this method, you redirect all HTTP/S traffic through the SV Lab server.

To learn using a HTTP/S proxy connector:

To make SSL certificate validation work, make sure the SV capture tool certificate has been added among the Trusted Root Certification
Authorities.
Make sure the SV Lab server is up and running.
Specify a port number. The SV Lab server exposes an intercepting HTTP/S proxy on this port. If you do not provide a value, the default port, 9000, is used.
Redirect all HTTP/S traffic to the SV Lab server by setting the system proxy or proxy used by JVM or application server to the port set in the previous step. For example, if the SV Lab server runs locally on the default port, use Address:localhost and Port:9000.
Run the capture utility with the following parameters:
```
sv-capture -p <SV Lab Server Port> 
           -as <Application Scenario Name>
           -r <http(s)://myserver.com/mypath> 
           -o <Output Folder>
```
The example above does not include the connector option -c HTTP_PROXY, since it is the default value.
1. Specify the SV Lab server port, upon which the SV Lab server listens for traffic. If you do not provide a value, the default port, 9000, is used.
2. Specify an application scenario name. If you do not provide a name, the default value, capture, is used.
3. Provide an endpoint URL as specific as possible, in order to filter
  the traffic that needs to be replaced with simulated messages.
4. Specify an output folder for the recorded application scenario.
Trigger the communication to be recorded and learned. Observe the messages being intercepted and displayed on the console output.
Press Enter to finish recording and trigger the learning process. The capture utility closes.
If necessary, reset the system proxy back to its original value. To allow the traffic to continue flowing through the SV Lab server, run the sv-capture tool again in HTTP endpoint discovery mode.

Tip

If you are behind a company proxy, you must set the -x parameter.
To check the recorded application scenario, run it in simulation mode.
The capture utility parameters used in recording are saved in the sv-lab.json file located in output directory. You can use the configuration file in subsequent learning or simulating sessions.

HTTP endpoint discovery mode

Note

Running Discovery mode is only relevant for the HTTP proxy connector. For details, see HTTP_PROXY vs. HTTP_GATEWAY.

When you run the SV capture utility in endpoint discovery mode, all endpoints discovered in HTTP/S traffic are sent to the console output. This allows you to view all of the endpoints involved, and select the one to virtualize. It also helps you verify that all traffic has been successfully redirected through the SV Lab server.

Resources accessed over the HTTP protocol are displayed with a complete URL, such as http://domain.server.com/path. For HTTPS connections, the path is left out, due to security considerations. The only exceptions are the virtual service endpoints hosted on servers hosting also endpoints specified in the sv-lab.json file, provided that you ran the utility with the -lab parameter.

To start a discovery, run the following command:

sv-capture -p 9000 -d

Use the -dd parameter instead of -d to display multiple occurrences of the same endpoint being accessed over time. When you use the -d parameter, each endpoint is only printed once.
The default value of the -p parameter is 9000. You can leave out this parameter when using the default value.

Learning using a virtual endpoint

This section describes how to record a session using a virtual endpoint using the capture utility.

To learn with a virtual endpoint:

To make SSL certificate validation work, make sure the SV capture tool certificate has been added among the Trusted Root Certification
Authorities.
Make sure the SV Lab server is up and running.
Select a virtual endpoint to replace the real endpoint inside the tested app's configuration. For example if the SV Lab server runs locally, you can replace a real endpoint http://myserver.com/mypath with a virtual one http://localhost:9000/mytest/mytestpath. Change the configuration file of the app under test/development to point to the virtual endpoint instead of the real one.
Run the capture utility with the following parameters:
```
sv-capture -c HTTP_GATEWAY
           -as <Application Scenario Name>
           -r <http(s)://myserver.com/mypath>
           -e <http(s)://localhost:9000/mytest/mytestpath>
           -o <Output Folder>
```
1. Specify a virtual endpoint -e, and the real endpoint -r. During recording the SV Lab server forwards the traffic it has intercepted on the virtual endpoint, to the real endpoint.
2. Specify an application scenario name. If you do not provide a name, the default value, capture, is used.
3. Provide an endpoint URL as specific as possible, in order to filter
  the traffic that needs to be replaced with simulated messages.
4. Specify an output folder for the recorded application scenario.
Trigger the communication to be recorded and learned. Observe the messages being intercepted and displayed on the console output.
Press Enter to finish recording and trigger the learning process. The capture utility closes.

Tip

To check the recorded application scenario, run it in simulation mode.
The capture utility parameters used in recording are saved in the sv-lab.json file located in output directory. You can use the configuration file in subsequent learning or simulating sessions.

HTTP_PROXY vs. HTTP_GATEWAY

There are two ways to record REST messages: HTTP_PROXY or HTTP_GATEWAY.

HTTP proxy

HTTP_PROXY is the default and preferred method for learning and simulation of REST services. With this method, you redirect all HTTP/S traffic through the SV Lab server, by setting the system proxy or proxy used by JVM or application server to the location at which the SV Lab server is listening.

Parameters:

-c HTTP_PROXY: Using the HTTP proxy connector.
-p <port number>: The port on which the SV Lab listens.
-r < endpoint URL>: The endpoint of the real service to record.
-x <proxy_address>: The proxy server, if you are behind a company proxy.

For details, see Learning using a proxy.

Pros	Cons
You don't need to have any control over the virtualized application or its clients, as long as you have control over your machine's or JVM proxy.	You must redirect all HTTP/S traffic to pass through the SV Lab server. If you are behind a company proxy, you must also set the proxy (using the -x parameter).

HTTP gateway

HTTP_GATEWAY is the preferred method if you want to avoid redirecting all traffic through the SV Lab server. To use this method, you must be able to change the config file of the app under development/test in order to redirect the calls to a virtual endpoint, instead of the real one.

Parameters:

-c HTTP_GATEWAY: Using the HTTP gateway connector.
-e <virtual endpoint>: The virtual endpoint including port number.
-r < endpoint URL>: The endpoint of the real service to record.

For details, see Learning using a virtual endpoint.

Pros	Cons
You don't have to change the system proxy, or make any other machine-wide changes. If you are behind a company proxy, you don't have to take any extra steps. Only the traffic through the real endpoint is redirected through the SV Lab server, while all other traffic stays intact.	You must modify the tested or developed app's config file in order to redirect the calls to a virtual endpoint (instead of the real one). Services with certain complex REST APIs won't work without manual update of learned models.

Reusing command line parameters stored in sv-lab.json

The SV capture utility parameters used in recording are saved in the sv-lab.json file located in the target directory.

To reuse them, use the -lab parameter: -lab <sv-lab.json file path> This has the same effect as manually specifying each parameter.

To change a value of any stored parameter, you must modify the content of the sv-lab.json file. The exception is the -f parameter that when specified, overrides the values in the file.

Tip

Create multiple copies of the saved sv-lab.json when you need to routinely switch the configuration parameters back and forth.

Recording behind a company proxy

To record and simulate traffic behind a company proxy, while using with the HTTP proxy connector, you must specify the company's proxy information. Use following syntax: -x <myproxy.mycomany.com:8080>. The SV Lab server forwards all the intercepted traffic towards the company proxy while recording it.

Note

If you keep the system proxy redirected to the SV Lab server after your session, your Internet connection may not work. Revert the system proxy back to the original value, or run the SV capture tool again in HTTP endpoint discovery mode.

Scenario simulation quick check

Once you have learned an application scenario, you can run it in simulation mode to check whether the learned scenarios work as expected. This is especially important before making any manual tweaks to the learned scenario code.

To run a check in simulation mode

To ensure a successful SSL certificate validation, make sure that the SV capture utility certificate has been added among the Trusted Root Certification Authorities.
Make sure the SV Lab server is up and running.
If you need to use the HTTP_PROXY connector, make sure the system proxy has been set up accordingly.
Run the capture utility:
```
sv-capture -lab <sv-lab.json File Path>
           -vsl <Application Scenarios Folder Path>
           -as <Application Scenario Name>
           -f NONE
```
1. Provide a path to the sv-lab.json file which contains the parameters used in recording.
2. Specify the folder containing files with the recorded application and service scenarios.
3. Specify the application scenario name that you want to simulate. If you do not provide a name, the default value, capture, is used.

Note

The -f NONE parameter forces the capture utility to simulate messages for all the deployed virtual services defined in sv-lab.json, instead of forwarding them to the real endpoint. Forwarding traffic is the default behavior of the SV capture utility.

Simulation without sv-lab.json

If you do not have access to the sv-lab.json file generated during recording, you can still test your application scenario in simulation mode, by explicitly providing the necessary parameters.

To run simulation using HTTP_PROXY

Specify the real endpoint and set the service mode to SIMULATE. You can omit -c HTTP_PROXY, since it is the default value.
Specify the folder containing files with the recorded application and service scenarios.
Specify the application scenario name that you want to simulate. If you do not provide a name, the default value, capture, is used.

sv-capture -r <http(s)://myserver.com/mypath> -m SIMULATE
           -vsl <Application Scenarios Folder Path>
           -as <Application Scenario Name> -c HTTP_PROXY

To run simulation using HTTP_GATEWAY

Specify the virtual endpoint, set the service mode to SIMULATE and choose the -c HTTP_GATEWAY connector.
Specify the folder containing files with the recorded application and service scenarios.
Specify the application scenario name that you want to simulate. If you do not provide a name, the default value, capture, is used.

sv-capture -e <http(s)://localhost:9000/mytest/mytestpath>
            -m SIMULATE -c HTTP_GATEWAY 
           -vsl <Application Scenarios Folder Path>
           -as <Application Scenario Name>

Learning HTTPs connections

We recommend that you add the SV capture utility certificate, sv-capture.pem, located in Lab/bin folder, to the Trusted Root Certification Authorities . No password is required.

When simulating messages over HTTPS, the SV Lab server generates its own SSL certificates for the virtual endpoint that mimic the certificate of real service. However, it appears as if it was issued by the "SV Capture Utility Certification Authority". This is not an actual authority. In order to avoid possible issues with certificate validation, add the fake authority's cartificate from sv-capture.pem to the Trusted Certification Authorities.

In Windows: Go to the Internet Options > Content tab. Click the Certificates button and select the Trusted Root Certification Authorities tab. Click Import, Next, and type the path to the sv-capture.pem file (when browsing, use the *.pem filter). Click Next several times and Finish. A successful import notification message box opens.

SV capture tool parameters

SV Capture Utility Parameter	Explanation
`-as,--application-scenario <arg>`	The application scenario name to record or simulate ("capture" by default)
`-c,--connector-type <arg>`	The connector type to use: `HTTP_PROXY` (default), `HTTP_GATEWAY`, or `MOBILE_API`
`-d,--discovery`	Run in HTTP endpoint discovery mode
`-dd,--discovery-all`	Run in verbose HTTP enpoint discovery mode, without endpoint duplicity removal
`-e,--endpoint-url <arg>`	The URL of the virtual endpoint to expose when using the `HTTP_GATEWAY` connector
`-f,--forward <arg>`	Override the service mode of the virtual service(s) specified in sv-lab.json. One of `ALL` (default), `NONE` or the `<virtualServiceId>`
`-fd,--force-deploy`	Force the deployment by undeploying an existing lab with the same ID if it already exists
`-h,--help`	Print out usage help
`-lab,--sv-lab-config-path <arg>`	The path to the sv-lab.json file, which defines virtual service endpoints and stores all parameters set during recording
`--log-level <arg>`	Set the log level: `TRACE`, `ALL`, `ERROR`, `INFO`, `FATAL`, `DEBUG`, `OFF`, or `WARN`
`-m,--service-mode <arg>`	Sets the Service Mode if no sv-lab.json available. One of `FORWARD` (default), `SIMULATE`, or `INVOKE`
`-ms,--mc-server-url <arg>`	Mobile Center endpoint in a `http(s)://hostname:port` format
`-ma,--mc-access-key <arg>`	Mobile Center execution access key
`-mi,--mc-phone-id <arg>`	Mobile Center mobile phone ID
`-mn,--mc-phone-name <arg>`	Mobile Center mobile phone name (alternative to ID)
`-mu,--mc-username <arg>`	Mobile Center username (when not using access key)
`-mp,--mc-password <arg>`	Mobile Center password (when not using access key)
`-o,--output <arg>`	The output folder to store the captured application and service scenario files
`-p,--capture-port <arg>`	TCP port where the SV Lab server should listen for traffic
`-r,--real-service <arg>`	The real REST service endpoint to be recorded and simulated
`-u,--sv-lab-url <arg>`	The SV Lab server API Endpoint URL
`-v,--verbose`	Show more information about processing.
`-vsl,--vsl-path <arg>`	The folder with the application and service scenario files to use for simulation
`-x,--outbound-proxy <arg>`	The original system `proxy:port`, before the traffic was redirected to the SV Lab server