NATS Controllers for Kubernetes (NACK)

Table of Contents

• JetStream Controller
  • Controller Modes
  • Getting Started
  • Managing Multiple NATS Systems and Accounts
  • Creating NATS Resources
  • Getting Started with Accounts
  • Local Development
• NATS Server Config Reloader
• NATS Boot Config

JetStream Controller

The JetStream controllers allows you to manage NATS JetStream resources via Kubernetes CRDs.

Controller Modes

NACK supports two controller modes with different capabilities:

Mode	Streams	Consumers	Key/Value	Object Store	Accounts
Legacy (default)	✅	✅	❌	❌	❌
Control-loop (--control-loop)	✅	✅	✅	✅	✅

Important: Key/Value stores and Object stores are only supported in control-loop mode. If you create KeyValue or ObjectStore resources without enabling control-loop mode, they will not be reconciled.

Resources managed by NACK controllers are expected to exclusively be managed by NACK, and configuration state will be enforced if mutated by an external client.

API Reference

The API reference documents all available CRD fields for Streams, Consumers, KeyValue, ObjectStore, and Account resources.

Getting started

Install with Helm:

helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm repo update

helm upgrade --install nats nats/nats \
  --set config.jetstream.enabled=true \
  --set config.jetstream.memoryStore.enabled=true \
  --set config.cluster.enabled=true --wait

helm upgrade --install nack nats/nack \
  --set jetstream.nats.url=nats://nats.default.svc.cluster.local:4222 --wait

(Optional) Enable Experimental controller-runtime Controllers

Note: The updated controllers will more reliably enforce resource state. If migrating from an older version of NACK, as long as all NATS resources are in-sync with NACK resources no modifications are expected.

The jetstream-controller logs will contain a diff of any changes the controller has made.

helm upgrade nack nats/nack \
  --set jetstream.nats.url=nats://nats.default.svc.cluster.local:4222 \
  --set jetstream.additionalArgs={--control-loop} --wait

Managing Multiple NATS Systems and Accounts

There are several approaches for managing multiple NATS Systems with NACK within one Kubernetes cluster. These options are not mutually exclusive.

1. Run Multiple Namespaced Controllers

You can run multiple NACK controllers on the same Kubernetes cluster. Add --set config.namespaced=true to your install flags or set namespaced: true in your values.yaml. When set, the controller will only reconcile resources within its own namespace.

helm upgrade --install nack nats/nack \
  --create-namespace --namespace nats \
  --set namespaced=true \
  --set jetstream.nats.url=nats://nats.nats.svc.cluster.local:4222 --wait

2. Use the Accounts Resource

The Accounts resource acts as a connection config for other resources. You may define multiple accounts for the same, or for distinct, NATS Systems.

---
apiVersion: jetstream.nats.io/v1beta2
kind: Account
metadata:
  name: a
spec:
  name: a
  creds:
    secret:
      name: account-a-creds
  servers:
    - nats://nats.nats-a.svc.cluster.local
---
apiVersion: jetstream.nats.io/v1beta2
kind: Account
metadata:
  name: b
spec:
  name: b
  creds:
    secret:
      name: account-b-creds
  servers:
    - nats://nats.nats-b.svc.cluster.local
---
apiVersion: jetstream.nats.io/v1beta2
kind: Stream
metadata:
  name: foo-a
spec:
  name: foo
  subjects: ["foo", "foo.>"]
  storage: file
  replicas: 3
  maxAge: 1h
  account: a
---
apiVersion: jetstream.nats.io/v1beta2
kind: Stream
metadata:
  name: foo-b
spec:
  name: foo
  subjects: ["foo", "foo.>"]
  storage: file
  replicas: 3
  maxAge: 1h
  account: b

The above manifests will define two Account resources, each pulling credentials from a Kubernetes secret. Account a is configured to use the NATS Cluster in namespace nats-a and Account b is configured to use the NATS Cluster in namespace nats-b. The NATS clusters do not need to be in Kubernetes, this is just an example.

This will also create an identical stream, foo, in each cluster. Note: The resource names, foo-a and foo-b, must be distinct to not conflict as Kubernetes resources, but the stream names themselves are both foo.

See more details in the Getting Started with Accounts section.

3. Define Connection Config in the CRD Manifest

You may define some connection options within the resource manifests directly. If not running in the newer --control-loop mode, set --crd-connect.

If running with --control-loop, resource-level connection config will always override any global config.

Note: The --crd-connect flag is not required if running with --control-loop.

helm upgrade nack nats/nack \
  --set jetstream.additionalArgs={--crd-connect} --wait

Example Stream:

apiVersion: jetstream.nats.io/v1beta2
kind: Stream
metadata:
  name: bar
spec:
  name: bar
  subjects: ["bar", "bar.>"]
  storage: file
  replicas: 3
  maxAge: 1h
  servers:
    - nats://nats.nats.svc.cluster.local:4222

Creating NATS Resources

Let’s create a stream and a couple of consumers:

---
apiVersion: jetstream.nats.io/v1beta2
kind: Stream
metadata:
  name: mystream
spec:
  name: mystream
  subjects: ["orders.*"]
  storage: memory
  maxAge: 1h
---
apiVersion: jetstream.nats.io/v1beta2
kind: Consumer
metadata:
  name: my-push-consumer
spec:
  streamName: mystream
  durableName: my-push-consumer
  deliverSubject: my-push-consumer.orders
  deliverPolicy: last
  ackPolicy: none
  replayPolicy: instant
---
apiVersion: jetstream.nats.io/v1beta2
kind: Consumer
metadata:
  name: my-pull-consumer
spec:
  streamName: mystream
  durableName: my-pull-consumer
  deliverPolicy: all
  filterSubject: orders.received
  maxDeliver: 20
  ackPolicy: explicit
---
# Note: KeyValue requires control-loop mode to be enabled
apiVersion: jetstream.nats.io/v1beta2
kind: KeyValue
metadata:
  name: my-key-value
spec:
  bucket: my-key-value
  history: 20
  storage: file
  maxBytes: 2048
  compression: true
---
# Note: ObjectStore requires control-loop mode to be enabled
apiVersion: jetstream.nats.io/v1beta2
kind: ObjectStore
metadata:
  name: my-object-store
spec:
  bucket: my-object-store
  storage: file
  replicas: 1
  maxBytes: 536870912 # 512 MB
  compression: true

# Create a stream.
$ kubectl apply -f https://raw.githubusercontent.com/nats-io/nack/main/deploy/examples/stream.yml

# Check if it was successfully created.
$ kubectl get streams
NAME       STATE     STREAM NAME   SUBJECTS
mystream   Ready     mystream      [orders.*]

# Create a push-based consumer
$ kubectl apply -f https://raw.githubusercontent.com/nats-io/nack/main/deploy/examples/consumer_push.yml

# Create a pull based consumer
$ kubectl apply -f https://raw.githubusercontent.com/nats-io/nack/main/deploy/examples/consumer_pull.yml

# Check if they were successfully created.
$ kubectl get consumers
NAME               STATE     STREAM     CONSUMER           ACK POLICY
my-pull-consumer   Ready     mystream   my-pull-consumer   explicit
my-push-consumer   Ready     mystream   my-push-consumer   none

# If you end up in an Errored state, run kubectl describe for more info.
#     kubectl describe streams mystream
#     kubectl describe consumers my-pull-consumer

Now we’re ready to use Streams and Consumers. Let’s start off with writing some data into mystream.

# Run nats-box that includes the NATS management utilities, and exec into it.
$ kubectl exec -it deployment/nats-box -- /bin/sh -l

# Publish a couple of messages from nats-box
nats-box:~$ nats pub orders.received "order 1"
nats-box:~$ nats pub orders.received "order 2"

First, we’ll read the data using a pull-based consumer.

From the above my-pull-consumer Consumer CRD, we have set the filterSubject of orders.received. You can double check with the following command:

$ kubectl get consumer my-pull-consumer -o jsonpath={.spec.filterSubject}
orders.received

So that’s the subject my-pull-consumer will pull messages from.

# Pull first message.
nats-box:~$ nats consumer next mystream my-pull-consumer
--- subject: orders.received / delivered: 1 / stream seq: 1 / consumer seq: 1

order 1

Acknowledged message

# Pull next message.
nats-box:~$ nats consumer next mystream my-pull-consumer
--- subject: orders.received / delivered: 1 / stream seq: 2 / consumer seq: 2

order 2

Acknowledged message

Next, let’s read data using a push-based consumer.

From the above my-push-consumer Consumer CRD, we have set the deliverSubject of my-push-consumer.orders, as you can confirm with the following command:

$ kubectl get consumer my-push-consumer -o jsonpath={.spec.deliverSubject}
my-push-consumer.orders

So pushed messages will arrive on that subject. This time all messages arrive automatically.

nats-box:~$ nats sub my-push-consumer.orders
17:57:24 Subscribing on my-push-consumer.orders
[#1] Received JetStream message: consumer: mystream > my-push-consumer / subject: orders.received /
delivered: 1 / consumer seq: 1 / stream seq: 1 / ack: false
order 1

[#2] Received JetStream message: consumer: mystream > my-push-consumer / subject: orders.received /
delivered: 1 / consumer seq: 2 / stream seq: 2 / ack: false
order 2

Getting Started with Accounts

You can create an Account resource with the following CRD. The Account resource can be used to specify server and TLS information.

Note The Account resource does not create or manage NATS accounts. It functions as a connection and authentication config for the managed resources.

The nsc tool can be used to manage your NATS account configuration on the server-side. You can find more details about NATS decentralized auth in the docs.

---
apiVersion: jetstream.nats.io/v1beta2
kind: Account
metadata:
  name: a
spec:
  name: a
  servers:
    - nats://nats:4222
  tls:
    secret:
      name: nack-a-tls
    ca: "ca.crt"
    cert: "tls.crt"
    key: "tls.key"

You can then link an Account to a Stream so that the Stream uses the Account information for its creation.

---
apiVersion: jetstream.nats.io/v1beta2
kind: Stream
metadata:
  name: foo
spec:
  name: foo
  subjects: ["foo", "foo.>"]
  storage: file
  replicas: 1
  account: a # <-- Create stream using account A information

The following is an example of how to get Accounts working with a custom NATS Server URL and TLS certificates.

# Install cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.17.0/cert-manager.yaml

# Install TLS certs
cd examples/secure

# Install certificate issuer
kubectl apply -f issuer.yaml

# Install account A cert
kubectl apply -f nack-a-client-tls.yaml

# Install server cert
kubectl apply -f server-tls.yaml

# Install nats-box cert
kubectl apply -f client-tls.yaml

# Install NATS cluster
helm upgrade --install -f nats-helm.yaml nats nats/nats

# Verify pods are healthy
kubectl get pods

# Install JetStream Controller from nack
helm upgrade --install nack nats/nack --set jetstream.enabled=true

# Verify pods are healthy
kubectl get pods

# Create account A resource
kubectl apply -f nack/nats-account-a.yaml

# Create stream using account A
kubectl apply -f nack/nats-stream-foo-a.yaml

# Create consumer using account A
kubectl apply -f nack/nats-consumer-bar-a.yaml

After Accounts, Streams, and Consumers are created, let’s log into the nats-box container to run the management CLI.

# Get container shell
kubectl exec -it deployment/nats-box -- /bin/sh -l

There should now be some Streams available, verify with nats command.

# List streams
nats stream ls

You can now publish messages on a Stream.

# Push message
nats pub foo hi

And pull messages from a Consumer.

# Pull message
nats consumer next foo bar

Local Development

# First, build the jetstream controller.
make jetstream-controller

# Next, run the controller like this
./jetstream-controller -kubeconfig ~/.kube/config -s nats://localhost:4222

# Pro tip: jetstream-controller uses klog just like kubectl or kube-apiserver.
# This means you can change the verbosity of logs with the -v flag.
#
# For example, this prints raw HTTP requests and responses.
#     ./jetstream-controller -v=10

# You'll probably want to start a local Jetstream-enabled NATS server, unless
# you use a public one.
nats-server -DV -js

Build Docker image

make jetstream-controller-docker ver=1.2.3

NATS Server Config Reloader

This is a sidecar that you can use to automatically reload your NATS Server configuration file.

Installing with Helm

For more information see the Chart repo.

helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm upgrade --install nats nats/nats

Configuring

reloader:
  enabled: true
  image: natsio/nats-server-config-reloader:0.16.1
  pullPolicy: IfNotPresent

Local Development

# First, build the config reloader.
make nats-server-config-reloader

# Next, run the reloader like this
./nats-server-config-reloader

Build Docker image

make nats-server-config-reloader-docker ver=1.2.3

NATS Boot Config

A helper utility used during NATS server pod initialization to generate and manage boot-time configuration.

Installing with Helm

For more information see the Chart repo.

helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm upgrade --install nats nats/nats

Configuring

bootconfig:
  image: natsio/nats-boot-config:0.16.1
  pullPolicy: IfNotPresent

Local Development

# First, build the project.
make nats-boot-config

# Next, run the project like this
./nats-boot-config

Build Docker image

make nats-boot-config-docker ver=1.2.3