GPATs

These are the Garden Performance Acceptance Tests.

About

These tests run as part of the garden-runcpipeline and will gate us from shipping a new release if they fail. With this in mind, the goal is to keep this test suite as minimal and quick as possible.

Metrics generated by the test suite are currently sent to Wavefront. This is achieved by wiring up the custom Wavefront Reporter into Ginkgo. In the future, should we wish to send metrics to other services, we can extend the suite by writing and wiring up a new custom reporter for that service.

Note that the reporter package used here is based on this reporter package from the cloudfoundry/benchmarkbbs repo.

Usage

In order to run the performance test suite you need to deploy garden and run the fololowing command (you will need the garden server ip as shown in bosh -d vms):

cd $HOME/workspace/garden-ci/directors/eden
fly -t garden-ci login -c https://garden.ci.cf-app.com/
GARDEN_ADDRESS="<garden-server-ip>" fly -t garden-ci execute \
  -c $HOME/workspace/garden-performance-acceptance-tests/ci/performance-tests.yml \
  -i gpats-main=$HOME/workspace/garden-performance-acceptance-tests

NB: This test suite will destroy ALL containers on the Garden server as part of the run. You have been warned.

Conditionally Expect Metrics

This suite is used both to gate releases via expectations on performance but also to provide metrics when thresholds are succeeded. To achieve this, metrics related expectations are wrapped in Conditionally() functions, to ensure they only fail tests when required. To turn off metrics related expectations, set the IGNORE_PERF_EXPECTATIONS environment variable.