param_runner

param_runner.sh is a minimalistic task batching / queueing tool built on top of etcd version 3, written as a bash script library.

In order to use param_runner, write your own task as a bash function that takes parameters to partition work item, then pass the function as an argument to param_runner::run. The following example illustrates the usage of this library end to end.

Installation

etcdctl commandline tool is required. See etcd document for instructions on getting etcd, which also bundles etcdctl. You can either put etcdctl in PATH, or path to etcdctl can be specified by --param_runner_etcdctl_bin flag.

Apart from bash, coreutils and etcdctl executable, the only external dependency is jq used for commandline json parsing. jq is widely available as binary package in most popular OS distributions. In Ubuntu, install jq by doing:

$ sudo apt-get install jq

Example

Suppose we have a simple addition calculating job defined (see the included example for the full example):

#!/bin/bash

source path/to/param_runner_etcd.sh || exit 1

FLAGS "$@" || exit 1
set -- "${FLAGS_ARGV[@]}"
extra_flags=( "$@" )

function run {
  echo "Task example $1: $2 + $3 = $(($2 + $3))"
  echo "${extra_flags[@]}"
}

param_runner::run run

Note that param_runner.sh sources shflags, and it is the wrapping script’s job to initialize FLAGS (after defining its own flags). This job takes three parameters, the name of task and two numbers whose sum will be an output of the job. Also note the extra_flags handling. This allows us to specify parameters at run time independent of the parameter sets. Now run the example:

$ ./example/calc_and_sleep_etcd.sh -- --some --extra=argument

You will see the following message, among other messages:

Listing /ls/test/home/$USER/param_runner/param to find any param set to work on

Note that /ls/test/home/$USER/param_runner is the default base path. This can be overridden with --param_runner_base_path flag.

Note that we use etcdctl version 3 API.

$ export ETCDCTL_API=3

Now we can enqueue some tasks by placing parameter sets under /param/:

$ for i in {1..10}; do
  etcdctl put /ls/test/home/$USER/param_runner/param/$i "${i}plusTen $i 10"
done

You will see the example program start processing the tasks:

Running with parameter set found at /ls/test/home/$USER/param_runner/param/1
Task example 1plusTen: 1 + 10 = 11
--some --extra=argument
Parameter set 1 succeeded

...etc

We can see the execution results by listing /result path:

$ etcdctl get --prefix /ls/test/home/$USER/param_runner/result
/ls/test/home/username/param_runner/result/20160901220004_SUCCESS_1_hostname_33783
1plusTen 1 10
/ls/test/home/username/param_runner/result/20160901220014_SUCCESS_10_hostname_33783
10plusTen 10 10
/ls/test/home/username/param_runner/result/20160901220025_SUCCESS_2_hostname_33783
2plusTen 2 10
/ls/test/home/username/param_runner/result/20160901220035_SUCCESS_3_hostname_33783
3plusTen 3 10
...etc

If there is any failure, you will see _FAIL_ instead of _SUCCESS_. The content of these files will be identical to the very parameter given.

Once we are done with the tasks, we can place poison pill so that the param runner will exit gracefully.

$ etcdctl put /ls/test/home/$USER/param_runner/poison_pill "."

The example program will now exit:

Poison pill found, exiting

Deployment with Docker

See the included example Docker file.

Disclaimer

This is not an official Google product.