API Recorder

Overview

API Recorder is a tool that captures the traffic between API clients and API providers. For APIs exposed over the HTTP protocol, it records each raw HTTP request and response into a pair of files.

Why is this useful?

definition

The recordings make it easy to create and configure API simulation simlets!

API Recorder captures HTTP/1.x traffic. Support for capturing HTTP/2 traffic will be provided in the future.

How It Works

API Recorder acts as a man-in-the-middle between API clients (like your application) and API providers (API dependencies). All requests and responses go through API Recorder, which relays them to the API providers and clients, respectively, after saving the traffic to files. What is recorded is the raw HTTP traffic: request line and response status line, header fields, body, and the original carriage return and line feed characters.

api-recorder-diagram

For HTTPS traffic, API Recorder terminates the TLS connection so it can capture the unencrypted traffic, and establishes a new TLS connection with the destination API provider.

When the HTTP body of a response is chunked and/or compressed, API Recorder decodes the content and then records it. The client still gets back the original, unchanged response that is chunked and/or compressed. The recorded response has the original Content-Encoding header (e.g. Content-Encoding: gzip) even if the content was decoded (e.g. decompressed). That helps in modeling the API so you know to configure simlets to use the same content encoding as the one for the real API.

Modes of Operation

API Recorder supports two modes of operation:

  • Proxy, and

  • Router.

Proxy Mode

When API Recorder is running in proxy mode, API clients do not have to change the URLs where they submit their requests. The HTTP client library they use has to be configured to use the API Recorder as an HTTP/S proxy. The API client is unaware that requests are sent to the API Recorder first which then relays them to the API provider’s endpoint.

Consult the documentation of your HTTP client for how to configure the use of HTTP/S proxy.

Router Mode

API Recorder in router mode routes the incoming requests to pre-configured host and port number for the API provider. An API client has to be configured to submit its requests to a running API Recorder and not to the API provider.

In the future, we may add support for routing tables to allow the routing of HTTP traffic to more than one host and port number with a single instance of API Recorder.

Support for TLS

More often than not nowadays, an API client communicates with API providers using HTTPS. That is - TLS-wrapped connection to secure the HTTP communications.

In either mode of API Recorder operation, an API client connects to API Recorder, which in turn establishes a connection with the API provider. When the connection has to be secured with TLS, API Recorder terminates the TLS from the API client and establishes a new TLS session with the API provider. The reason is that API Recorder must be able to get to the plain, unencrypted HTTP traffic in order to record it.

API Recorder dynamically detects HTTPS calls and does the necessary to complete the TLS handshake to establish a TLS connection. This requires an X.509 Certificate and its Private Key to be configured in API Recorder. If they are not specified at startup time, API Recorder will use the self-signed Certificate and Private Key bundled with the distribution.

API Recorder supports key stores in the Java KeyStore format (.jks). The cert and its corresponding Private Key have to be imported into a Java KeyStore and its location passed in as command line argument (see below).

Support for other formats, as .pem, and mutual authentication are planned for the future.

How to Use API Recorder

Go to the API Simulator’s bin directory (replace the version, 1.6, and the actual location of the install as appropriate):

cd "/path/to/apisimulator/apisimulator-http-1.6/bin"

It is recommended to add the path to the PATH environment variable - that makes it easier to start and stop API Recorder from any directory.

To start an API Recorder instance from the bin directory, execute the commands as described below:

Linux:

./apirecorder start <arguments>

If desired, you can background the process by appending &.

Windows:

apirecorder start <arguments>

If desired, you can launch API Recorder in a separate command prompt window with something like this:

start "API Recorder" /min cmd /c apirecorder start <arguments>

Notice that if for some reason there are errors when starting API Recorder (e.g. mistyped simulation directory) and logging is directed to the console (stdout), it may be difficult to troubleshoot because the log info will be lost when the new command prompt window is closed automatically due to the error.

To shut down API Recorder, use stop.

Linux:

./apirecorder stop <arguments>

Windows:

apirecorder stop <arguments>

On either platform, usually <arguments> are not needed when stopping an API Recorder instance unless its admin server is listening on a port other than the default.

Command-line Arguments

Argument Description

-m <arg>

Mode of capturing traffic. Expected value is proxy or router. Defaults to proxy if not specified.

-d <arg>

Directory destination for the recordings. API Recorder will use the current directory ., if this argument isn’t configured.

-h <arg>

Recorder host. Useful when there are multiple network interfaces.

-H <arg>

Remote host name or IP address when in router mode.

-p <arg>

API Recorder will listen on this local port number for connections. Required when starting API Simulator.

-P <arg>

Port number for the remote server when in router mode. The argument is optional and if not specified will default to 80 for HTTP and 443 for HTTPS requests.

-admin_host <arg>

Admin server host. Useful when there are multiple network interfaces.

-admin_key <arg>

Value of the key/passphrase, if the admin server is secured.

-admin_port <arg>

Admin server port number. The default is 8881.

-http_proxy <arg>

Proxy to use when calling the destination API provider over HTTP. See also Proxy Configuration.

-https_proxy <arg>

Proxy to use when calling the destination API provider over HTTPS. See also Proxy Configuration.

-no_proxy <arg>

Comma-delimited list of domains and IP addresses to exclude from proxying when calling the destination API provider over HTTP or HTTPS. See also Proxy Configuration.

-jks_file <arg>

Java KeyStore (JKS) file containing the TLS cert(s) to use. classpath:<KeyStoreFileName> will cause the file to be loaded from the classpath.

-jks_password <arg>

Password to access the Java KeyStore (JKS).

-key_password <arg>

Password for the key.

-tls <arg>

Optional argument. Its value denotes the format for the certificate and key - pem or jks (Java KeyStore) - to use for TLS. Currently pem is not supported.

Environment Variables

The following optional environment variables control various aspects of configuring API Recorder:

Environment Variable Description

APIRECORDER_JAVA

Makes it possible to start API Recorder using a Java installation different from the one JAVA_HOME points to.

APIRECORDER_HEAP_SIZE

Maximum and/or minimum JVM heap size and other memory-related settings.

Linux, macOS:

export APIRECORDER_HEAP_SIZE="-Xms256m -Xmx256m"

Windows:

set "APIRECORDER_HEAP_SIZE=-Xms256m -Xmx256m"

APIRECORDER_OPTS

Other JVM options and arguments except heap size. For example, Garbage Collector settings.

APIRECORDER_LOG

The value determines the logging destination. Valid values for are: stdout for writing to the console; file for writing to a local file; or empty string to disable logging.

APIRECORDER_LOG_FILE

Fully qualified path and file name of logging configuration file to use.

APIRECORDER_LOG_PATH

Path to the directory for the log files. The log files are named apirecorder.log and apirecorder-stop.log for when running and stopping API Recorder, respectively.

http_proxy

Proxy to use when calling the destination API provider over HTTP. See also Proxy Configuration.

HTTP_PROXY

Proxy to use calling the destination API provider over HTTP, if http_proxy isn’t configured. See also Proxy Configuration.

https_proxy

Proxy to use when calling the destination API provider over HTTPS. See also Proxy Configuration.

HTTPS_PROXY

Proxy to use when calling the destination API provider over HTTPS, if https_proxy isn’t configured. See also Proxy Configuration.

no_proxy

Comma-delimited list of domains and IP addresses to exclude from proxying when calling the destination API provider over HTTP or HTTPS. See also Proxy Configuration.

NO_PROXY

Comma-delimited list of domains and IP addresses to exclude from proxying when calling the destination API provider over HTTP or HTTPS, if no_proxy isn’t configured. See also Proxy Configuration.

Proxy Configuration

In some environments, egress traffic to external API providers is always routed through a forward proxy. API Recorder can be configured to use such proxy when relaying requests to API providers over HTTP and HTTPS. For that to work, API Recorder must be running in proxy mode. This way, requests sent to API Recorder will contain the actual destination address.

The format for a proxy configuration value is [user[:password]@]host:port, where [] denotes an optional part in the configuration.

API Recorder uses hierarchical lookup to configure HTTP or HTTPS proxy. Here are the steps for HTTPS proxy - configuring HTTP proxy follows similar steps:

  1. Look for program argument https_proxy

  2. If https_proxy program argument was not found or it was empty, look for JVM -D argument apisimulator.proxy.https

  3. If JVM argument was not found or it was empty, look for environment variable https_proxy

  4. If environment variable https_proxy was not found or it was empty, look for environment variable HTTPS_PROXY

If accessing both HTTP and HTTPS URLs, then explicitly configure both HTTP and HTTPS proxies even if they are the same proxy.

Examples

Using program arguments, configure different proxies for HTTPS and HTTP requests:

-https_proxy proxy.example.com:4888 -http_proxy proxy.example.com:8888

Using program arguments, configure the same proxy for both HTTPS and HTTP requests:

-https_proxy proxy.example.com:8443 -http_proxy proxy.example.com:8443

No Proxy

Different tools have different rules for "no proxy" syntax.

API Recorder accepts a comma-delimited list of domains and IP addresses. The only wildcard no_proxy and NO_PROXY accept is a single * character, which matches "all hosts"; that effectively disables the proxy.

API Recorder does not accept * as a wildcard attached to a domain suffix. For example, this works:

no_proxy=.example.com

However, this does not work:

no_proxy=*.example.com

The configuration

no_proxy=.example.com

will match example.com and www.example.com.

Port numbers and CIDR ranges in a "no proxy" configuration are not supported.

API Recorder Examples

Go to the API Simulator’s bin directory (replace the version, 1.6, and the actual location of the install as appropriate):

cd "/apisimulator/apisimulator-http-1.6/bin"

Proxy Mode

Start an API Recorder instance on port 8880 and configure the directory where the traffic recordings will be saved. This example reuses the logs directory but usually you would specify your own location:

Linux:

./apirecorder start -p 8880 -d "/apisimulator/apisimulator-http-1.6/logs" &

Windows:

apirecorder start -p 8880 -d "/apisimulator/apisimulator-http-1.6/logs"

Notice that we did not specify -m argument so the recorder by default will run in proxy mode. The internal admin server by default will listen on port 8881.

Let’s submit a request using cURL to get a jQuery UI CSS file that is hosted by Google. The --proxy option tells cURL to use our recorder as a proxy:

curl --proxy localhost:8880 -X GET "http://ajax.googleapis.com/ajax/libs/jqueryui/1.12.0/themes/smoothness/jquery-ui.css"

Look in the directory with the recordings - there should be two new files created for the raw request (input) and response (output).

Let’s now submit a request using cURL to get the same jQuery UI CSS file as above but this time we will use https instead of http:

curl --proxy localhost:8880 -X GET "https://ajax.googleapis.com/ajax/libs/jqueryui/1.12.0/themes/smoothness/jquery-ui.css" -k

Notice the -k option - it tells cURL not to validate the certificate. Reason being - API Recorder was started without configuring a key store with Certificate and Private Key issued by a trusted CA (Certificate Authority) so it is using a self-signed Certificate bundled with the distribution.

The directory with the recordings should contain two new files with the actual raw request (input) to ajax.googleapis.com and its response (output) when accessed over HTTPS.

Stop the API Recorder before continuing with another example.

Router Mode

Use the -m argument to start API Recorder as a router:

Linux:

./apirecorder start -p 8880 -d "/apisimulator/apisimulator-http-1.6/logs" -m router -H ajax.googleapis.com -P 80 &

Windows:

apirecorder start -p 8880 -d "/apisimulator/apisimulator-http-1.6/logs" -m router -H ajax.googleapis.com -P 80

The command tells API Recorder to route to ajax.googleapis.com:80 any request it receives.

Let’s now submit a request using cURL to fetch the jQuery UI CSS file as above but this time send the request to the API Recorder running on localhost, port 8880:

curl -X GET "http://localhost:8880/ajax/libs/jqueryui/1.12.0/themes/smoothness/jquery-ui.css"

The directory with the recordings should have two new files - one for the request (input) and one for the response (output).

Stop the API Recorder before continuing with another example.

Proxy Configuration

This assumes that all egress traffic for both HTTP and HTTPS is forced to go through a proxy proxy.example.com, port 8888. Requests to localhost, 127.0.0.1, and google.com and all its subdomains won’t be proxied.

Start an API Recorder instance on port 8880 and configure the directory where the traffic recordings will be saved. This example reuses the logs directory but usually you would specify your own location:

Linux:

./apirecorder start -p 8880 -d "/apisimulator/apisimulator-http-1.6/logs" -https_proxy proxy.example.com:8888 -http_proxy proxy.example.com:8888 -no_proxy "localhost,127.0.0.1,.google.com" &

Windows:

apirecorder start -p 8880 -d "/apisimulator/apisimulator-http-1.6/logs" -https_proxy proxy.example.com:8888 -http_proxy proxy.example.com:8888 -no_proxy "localhost,127.0.0.1,.google.com"

Notice that for this example we did not specify -m argument so the recorder by default will run in proxy mode but the concepts in this example apply to API Recorder running in router mode as well.

Let’s execute this HTTPS request:

curl -k --proxy localhost:8880 -X GET "https://www.google-analytics.com/analytics.js" -H "Accept-Encoding: gzip,deflate"

The --proxy option tells cURL to use our API Recorder, running on localhost port 8880, as a proxy. The -k option is to configure cURL not to validate the proxy certificate.

By passing Accept-Encoding: gzip,deflate header, we advertise that our client can accept compressed response. If the server supports it, the HTTP body in the response will be compressed and the result printed on the screen will be that compressed HTTP body. The recorded response, however, will have the uncompressed content.

Similarly, we can execute the same request but over HTTP:

curl -k --proxy localhost:8880 -X GET "http://www.google-analytics.com/analytics.js" -H "Accept-Encoding: gzip,deflate"

The example above demonstrated two features of API Recorder - configuring and using (forward) proxy, and dealing and capturing compressed responses. They can be configured and used independently of each other.

Stop the API Recorder before continuing with another example.


We would love to hear your feedback! Shoot us an email to [feedback at APISimulator.com] about anything that is on your mind.

Happy API Simulating!