cf-deployment-concourse-tasks

This repo contains concourse tasks for use with cf-deployment. If you are trying to deploy to IAAS, you may wish to check the Deployment Guide. Each task is in its own directory. A task will generally be composed of a task.yml to be referenced in pipelines, a supporting task file, and a Dockerfile. The Dockerfile is built and pushed to Dockerhub regularly in CI maintained by the CF Release Integration team here.

It should be clear how to use each task from the task.yml and the description below. If you find that it is not, please contact the Release Integration team in our Slack channel dedicated to supporting users of cf-deployment. Alternatively, you can open an issue.

Versioning of this repo

Development updates to the repo are made to the master branch, so untested or backwards incompatible changes may be present there. Once changes have been tested and all stories accepted, we add new version tags such as v1.6 to the approprate commit.

We use a bare-bones type of semantic versioning for this repo. Backwards incompatible changes warrant a major version bump (e.g. v1.6 to v2.0), while other changes will simply add a minor version bump (e.g. v2.0 to v2.1).

In Concourse, you can pretty easily lock to a major version, meaning that your pipeline will take minor (i.e. backwards compatible) changes only. Here’s an example from our nats release pipeline:

- name: cf-deployment-concourse-tasks
  type: git
  source:
    branch: main
    uri: https://github.com/cloudfoundry/cf-deployment-concourse-tasks.git
    tag_filter: v3.*

When you’re ready to take the backwards incompatible changes, you can take any necessary manual steps to upgrade, and then change the major version in your pipeline configuration.

Tasks

Tasks are listed here alphabetically, along with a brief description meant to be used alongside the task.yml within each task directory to understand the tasks’ purpose, interface, and options. Each title is also a link to the appropriate task.yml.

bbl-destroy

This destroys the director and infrastructure created by bbl. Debug output is written to bbl_destroy.txt to help debug failures in this task.

bbl-up

This uses bbl to create your infrastructure and deploy a BOSH director. Debug output is written to bbl_plan.txt and bbl_up.txt to help debug failures in this task. This task requires a certificate and key (unless you are bbling up a bosh-lite environment) which can be generated using the commands specified here.

bosh-cleanup

This performs a BOSH cleanup which is necessary from time to time to avoid running out of space.

bosh-delete-deployment

This deletes a BOSH deployment. If you want to delete all of the available BOSH deployments you can set the DELETE_ALL_DEPLOYMENTS flag to true.

bosh-deploy

This performs a BOSH upload-stemcell and BOSH deployment. Optionally, operations files may be applied to the deployment manifest.

It’s also configurable to regenerate deployment credentials on each deployment though this is not the default behavior. This is helpful for testing changes to variable generation, but is only expected to work with fresh deployments. This also automatically uploads the stemcells present in deployment. If you’re deploying to bosh-lite environment you need to set the BOSH_LITE flag to true so the task uploads the correct stemcell(s).

bosh-deploy-with-created-release

This creates and applies an additional operations file to cf-deployment.yml, which causes BOSH to create, upload, and use a dev release from the provided release folder in place of the version specified in cf-deployment.yml. This is useful for testing an upstream component. Otherwise identical to the bosh-deploy task above.

bosh-deploy-with-updated-release-submodule

This takes as input a concourse resource for the submodule version bumped when creating a dev release from the provided release folder. Otherwise identical to the bosh-deploy-with-created-release task above.

bosh-upload-stemcells

This uploads stemcell(s) associated with the manifest and/or ops files provided. This task can be used to upload stemcells within jobs that do not contain a bosh-deploy* task (which handles uploading stemcells as well as executing the bosh deployment).

collect-ops-files

This collects two sets of operations files. The first set is the “base” set, to which the second (“new”) set is added.

If there is a name conflict, the operations file from the second (“new”) set wins.

rotate-ca-certificates

This task performs a three-step CA certificate rotation according to the CredHub CA Rotation documentation. It can be used to renew all certificates in a long-lived CF deployment.

The cf-deployment default list of certificates is specified in the rotate-ca-certificates-task-yaml. If you use a non-default configuration (e.g. with Postgres instead of MySQL), you must adapt this list.

After each step, all BOSH VMs must be redeployed to update the certificates. This can be done with a simple helper task that calls the following BOSH commands:

  bosh -d cf manifest > manifest.yml
  bosh -d cf --non-interactive deploy manifest.yml

So the whole pipeline would consist of these jobs:

flowchart LR;
    A[rotate-step-1] --> B;
    B[redeploy] --> C;
    C[rotate-step-2] --> D;
    D[redeploy] --> E;
    E[rotate-step-3] --> F;
    F[redeploy];

The pipeline can be triggered by a time schedule e.g. weekly. The “rotate” jobs are idempotent and can be retriggered in case of failure (but make sure to preserve the order).