Boulder Release Process Docs & Demo

A repo for documenting and demonstrating the Boulder release process used to move code from Github to staging and production.

Goals

When we need to do a hotfix deploy, we can easily do one that contains only the fixes needed for hotfix, regardless of what changes have been committed in the meantime.
Any code we deploy meets our standard code review process and has passed all automated tests.
Any commit we deploy is eventually made available in the public boulder repo.
The hotfix release process differs as little as possible from the scheduled release process.

Release Schedule

Boulder developers make a new release at the beginning of each week, typically by 10am PST Monday. Operations deploys the new release to the staging environment on Tuesday, typically by 2pm PST. If there have been no issues discovered with the release from its time in staging, then on Thursday the operations team deploys the release to the production environment.

Holidays, unexpected bugs, and other resource constraints may affect the above schedule and result in staging or production updates being skipped. It should be considered a guideline for normal releases but not a strict contract.

Making a Release

Regular Releases

For regular staging releases, each week the Boulder dev team will tag the current main commit with the current date, making a tag prefixed by "release-" (e.g. release-2018-01-22). The tag name will always be the date the release is tagged, not the date that it is pushed to staging or production. If multiple releases are tagged on the same day the tag will be given an incrementing letter suffix (e.g. release-2018-01-22a)

Hotfix Releases

When main is clean

When main is clean, and has no commits that we do not wish to include in the hotfix release, then the regular release process is used. The hotfix lands in main after the normal review process and a dev tags a new release per “Regular Releases”.

When main is dirty

When main is dirty, and has commits that are unrelated to the hotfix and too risky to ship to prod we need to use an adjusted hotfix process. In this case dev team will create a branch based on the current release tag prefixed by release-branch-. Pull requests with the hotfix code will be made against main, where they will be reviewed and merged as normal. Then the resulting commits will be cherry-picked and used to create PRs against the release branch, where they will be re-reviewed and have CI re-run. Once the neccessary changes have been merged to the release branch dev team will create a new hotfix release tag based on that branch.

Deploying Releases

When doing a release, Ops’ tooling will check that:

Travis shows that tests have passed for the commit at the planned release tag.
The planned release tag is an ancestor of the current main on GitHub OR the planned release tag is equal to the head of a branch named release-branch-XXX, and that branch contains a release-tagged commit, and all commits since then have followed the Boulder review process.
Each commit on the hotfix branch references a PR, and that PR was reviewed, and the PR page shows the final merged commit id.

Prerequisites

You must have a GPG key with signing capability:
- Checking for existing GPG keys
If you don’t have a GPG key with signing capability, create one:
- Generating a new local GPG key
- Generating a new Yubikey GPG key
The signing GPG key must be added to your GitHub account:
- Adding a new GPG key to your GitHub account
git may need to be configured to call the correct GPG binary:
- The default: git config --global gpg.program gpg is correct for most Linux platforms
- On macOS and some Linux platforms: git config --global gpg.program gpg2 is correct
git must be configured to use the correct GPG key:
- Telling Git about your GPG key
Understand the process for signing tags

Example

On 2018-01-30, @cpu submits a pull request, proposing a feature to be merged to main.
@jsha reviews and approves the PR. The corresponding travis build passes, and so @cpu merges the PR into main.
Later that day, @cpu cuts a release by making a release-2018-01-30 tag from the tip of main at the time of tagging.
Ops deploys release-2018-01-30 some time later.
After deploying release-2018-01-30 a critical bug is discovered! Since main is clean, that is, has no commits since release-2018-01-30 we don’t want to ship with a hotfix, @cpu quickly prepares a hotfix PR to propose a hotfix change be merged into main.
@jsha reviews and approves the hotfix PR. The corresponding travis build passes, and so @cpu merges the hotfix PR into main.
With the main branch ready with a hotfix, @cpu cuts a hotfix release by making a release-2018-01-30a tag from the tip of main at the time of tagging.
Ops deploys release-2018-01-30a, fixing the bug
On 2018-01-31, @cpu submits a pull request to propose a new feature be merged into main.
Later that day @jsha reviews and approves the feature PR. The corresponding travis build passes, and so @cpu merges the PR.

Later on 2018-01-31 its discovered that the bug in release-2018-01-30, which was hotfixed with release-2018-01-30a is still broken! Uh oh! Since main is dirty, and contains a feature commit not present in release-2018-01-30 or release-2018-01-30a the hotfix must be prepared in a separate branch. @cpu starts a new branch release-branch-2018-01-30a by checking out release-2018-01-30a and pushing it as the release branch.
In a new branch off of release-branch-2018-01-30a @cpu creates a hotfix and creates a pull request proposing the hotfix be merged into release-branch-2018-01-30a.
@rolandshoemaker reviews and approves the hotfix PR. The corresponding travis build passed, and so @cpu merges the PR into release-branch-2018-01-30a.
@cpu then cuts a new release from release-branch-2018-01-30a by making a release-2018-01-31 tag from the tip of release-branch-2018-01-30a at the time of tagging.
Ops deploys release-2018-01-31, fixing the bug again without having to introduce unrelated code from main