Platforms

Kubernetes

Run a production control plane on Kubernetes as a StatefulSet with one persistent volume per instance.

This guide runs three control plane instances as a StatefulSet. Each pod gets its own persistent volume, and all three share one bucket.

Requirements

A Kubernetes cluster with at least three Kubernetes nodes, so each instance can land on its own machine
A StorageClass backed by block storage, such as gp3 on EKS or pd-ssd on GKE. Network file systems such as NFS or EFS are not supported.
An object storage bucket, with credentials that can read, write, list and delete in it.

Pods do not get raw NVMe disks, so a Kubernetes cluster will not match the published latency numbers. See Benchmarking.

Steps

Create the secret

kubectl create namespace rivet
kubectl -n rivet create secret generic rivet \
  --from-literal=rivet-token="$(openssl rand -hex 32)" \
  --from-literal=s3-access-key-id="AKIA..." \
  --from-literal=s3-secret-access-key="..."

namespace/rivet created
secret/rivet created

Apply the manifests

Save this as rivet.yaml, set the bucket name and region, and apply it.

apiVersion: v1
kind: Service
metadata:
  name: rivet
  namespace: rivet
spec:
  selector: { app: rivet }
  ports:
    - { name: http, port: 6420 }
---
apiVersion: v1
kind: Service
metadata:
  name: rivet-peer
  namespace: rivet
spec:
  clusterIP: None
  selector: { app: rivet }
  ports:
    - { name: peer, port: 6421 }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rivet
  namespace: rivet
spec:
  serviceName: rivet-peer
  replicas: 3
  podManagementPolicy: Parallel
  selector:
    matchLabels: { app: rivet }
  template:
    metadata:
      labels: { app: rivet }
    spec:
      terminationGracePeriodSeconds: 660
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels: { app: rivet }
              topologyKey: kubernetes.io/hostname
      containers:
        - name: rivet
          image: rivetdev/engine:2.3.31
          ports:
            - { name: http, containerPort: 6420 }
            - { name: peer, containerPort: 6421 }
            - { name: metrics, containerPort: 6430 }
          env:
            - name: POD_IP
              valueFrom: { fieldRef: { fieldPath: status.podIP } }
            - name: RIVET__API_PEER__ADVERTISE_ADDRESS
              value: "$(POD_IP):6421"
            - name: RIVET__DEPOT__LOG_DIRECTORY
              value: /var/lib/rivet/log
            - name: RIVET__OBJECT_STORAGE__PRIMARY__S3__BUCKET
              value: my-rivet-bucket
            - name: RIVET__OBJECT_STORAGE__PRIMARY__S3__REGION
              value: us-east-1
            - name: RIVET__OBJECT_STORAGE__PRIMARY__S3__ACCESS_KEY_ID
              valueFrom: { secretKeyRef: { name: rivet, key: s3-access-key-id } }
            - name: RIVET__OBJECT_STORAGE__PRIMARY__S3__SECRET_ACCESS_KEY
              valueFrom: { secretKeyRef: { name: rivet, key: s3-secret-access-key } }
            - name: RIVET__AUTH__ADMIN_TOKEN
              valueFrom: { secretKeyRef: { name: rivet, key: rivet-token } }
          readinessProbe:
            httpGet: { path: /health, port: 6420 }
          volumeMounts:
            - { name: log, mountPath: /var/lib/rivet/log }
  volumeClaimTemplates:
    - metadata:
        name: log
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Gi

kubectl apply -f rivet.yaml
kubectl -n rivet rollout status statefulset/rivet

partitioned roll out complete: 3 new pods have been updated...

The manifest uses an S3 bucket. See Storage for other object storage backends.

Check that the instances are serving

On first start the instances place their storage on each other automatically, which takes about a minute after the last pod is ready. The health route shows that an instance is serving requests.

kubectl -n rivet port-forward svc/rivet 6420:6420 &
curl http://localhost:6420/health

{"runtime":"engine","status":"ok","version":"2.3.31",...}

Expose port 6420

Put a load balancer or Ingress in front of the rivet Service and terminate TLS there. Raise its idle timeout to one hour, or it will cut the long-lived connections that nodes and clients hold open. See Ports & TLS.

Connect your nodes

Create the default node pool, because the control plane refuses a node whose pool does not exist. See Nodes & Node Pools.

export RIVET_API=https://rivet.example.com
export RIVET_TOKEN="$(kubectl -n rivet get secret rivet -o jsonpath='{.data.rivet-token}' | base64 -d)"
curl -X PUT "$RIVET_API/pools/default?namespace=default" -H "Authorization: Bearer $RIVET_TOKEN" -H 'Content-Type: application/json' -d '{}'

{"pool":{"name":"default","drain_on_version_upgrade":false,"actor_eviction_delay":0,"actor_eviction_period":0,"actor_eviction_rate":1.0,"metadata":null}}

Then set RIVET_ENDPOINT on your app to the public URL of the load balancer, with your Rivet token. See Deploy nodes.

RIVET_ENDPOINT=https://default:<rivet-token>@rivet.example.com

Keep port 6421 private

Instances talk to each other on port 6421 with no authentication. Allow it only between control plane pods:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: rivet-peer
  namespace: rivet
spec:
  podSelector:
    matchLabels: { app: rivet }
  policyTypes: ["Ingress"]
  ingress:
    - ports: [{ port: 6420 }]
    - from:
        - podSelector: { matchLabels: { app: rivet } }
      ports: [{ port: 6421 }]

Add a rule for port 6430 if Prometheus scrapes the instances.

Scaling

Add instances by raising replicas. The control plane moves storage onto new instances on its own. Never delete a PersistentVolumeClaim of a running cluster to free space: the instance that owned it is replaced, and its replicas are rebuilt from the other instances.