s2n-netbench

An efficiency, performance, and correctness analysis tool for transport protocols.

Why does this exist?

There are many transport protocols and several implementations of each. This tool exists to provide users with the ability to perform a direct comparison and decide the best implementation for their workloads.

Here are a few examples of questions that s2n-netbench aims to answer:

• What is the cost of encrypting traffic?
  • How much more capacity will I need when deploying this?
• What transport protocol performs best
  • in a data center?
  • in networks with high packet loss?
  • in networks with high latency?
  • with many concurrent, multiplexed streams?
• Which implementation of “X” protocol is best for my workload?
• What is the optimal configuration of the transport’s settings for my workload?
• How does certificate chain length affect handshake throughput?
• Is implementation “X” interoperable with implementation “Y” of “Z” protocol?

Quickstart

A basic use of s2n-netbench is demonstrated in the netbench-run.sh script. This script will

• compile all necessary s2n-netbench utilities
• generate scenario files
• execute the request-response.json scenario using s2n-quic and s2n-tls drivers
• execute the connect.json scenario using s2n-quic and s2n-tls drivers
• collect statistics from the drivers using netbench-collector
• generate a report in the ./target/netbench/report directory

From the main netbench folder, run the following commands

./scripts/netbench-run
cd target/netbench/report
python3 -m http.server 9000

Then navigate to localhost:9000 in a browser to view the netbench results.

Note that this script does not support bpftrace, as it runs without any of the elevated permissions required for bpf programs.

How it works

netbench-scenarios

netbench provides tools to write scenarios that describe application workloads. An example of a scenario is a simple request/response pattern between a client and server:

use netbench_scenario::prelude::*;

config!({
    /// The size of the client's request to the server
    let request_size: Byte = 1.kilobytes();

    /// The size of the server's response to the client
    let response_size: Byte = 10.megabytes();
});

pub fn scenario(config: Config) -> Scenario {
    let Config {
        request_size,
        response_size,
    } = config;

    Scenario::build(|scenario| {
        let server = scenario.create_server();

        scenario.create_client(|client| {
            client.connect_to(server, |conn| {
                conn.open_bidirectional_stream(
                    |local| {
                        local.send(request_size);
                        local.receive(response_size);
                    },
                    |remote| {
                        remote.receive(request_size);
                        remote.send(response_size);
                    },
                );
            });
        });
    })
}

This scenario generates a json file of instructions. These instructions are protocol and language independent, which means they can easily be executed by a “netbench driver”, written in any language or runtime.

netbench-driver

Netbench drivers are responsible for executing netbench scenarios. Each transport protocol has a client and server implementation. Each of these implementations is a self-container binary that consumes a scenario.json file. Implemented drivers include:

• TCP
• native-tls
  • OpenSSL on Linux
  • Secure Transport on macOS
  • SChannel on Windows
• s2n-quic
• s2n-tls

netbench-collector

Driver metrics are collected with the netbench-collector utility. There are two implementation of this available - a generic utility and a bpftrace utility. The generic utility uses the proc fs to gather information about the process, while the bpftrace implementation is able to collect a wider variety of statistics through ebpf probes.

The collector binary takes a netbench-driver as an argument. The driver binary is spawned as a child process. The collector will continuously gather metrics from the driver and emit those metrics to stdout.

netbench-cli

netbench-cli is used to visualize the results of the scenarios. This reports use vega which is “a declarative format for creating, saving, and sharing visualization designs”.

report is used to generate individual .json reports. These can be visualized by pasting them into the vega editor.

report-tree is used to to generate a human-readable .html report. Given a directory structure like the following

request-response/ # scenario
├─ tls/ # driver
│  ├─ client.json
│  ├─ server.json
├─ quic/
   ├─ client.json
   ├─ server.json

report-tree will generate the individual reports and package them into a human readable index.html file that can be used to view graphs of the results.

A sample report can be found here.

Note that you will not be able to open the report directly since the report relies on the jsdelivr cdn. This request will fail when the URL is a local file scheme with a CORS request not HTTP error.

To get around this, use a local server.

# assuming the report is in ./report
cd report
# start a local server on port 9000
python3 -m http.server 9000

In a browser, navigate to localhost:9000 to view the netbench report.