Kubernetes and KEDA

Phase 3 adds a Kubernetes mode alongside Docker Compose. The same binaries and images run in both environments; only orchestration and the autoscaler actuator change.

How KEDA fits in test-scale

test-scale splits autoscaling into two layers:

Layer	Responsibility	Docker Compose	Kubernetes
Brain	Decide how many workers are needed (ceil(target_rps / per_worker_rps))	Shared	Shared
Scaler	Apply that count to the runtime	DockerScaler → docker compose scale	KedaScaler → Prometheus gauge → KEDA → Deployment

KEDA does not decide capacity. The control plane Brain computes the number (e.g. 2000 RPS ÷ 500 RPS/worker = 4 workers). KEDA is the Kubernetes actuator that turns that decision into running pods.

Why not CPU-based HPA?

Load-test workers often sit idle, then ramp hard. CPU-based Horizontal Pod Autoscaler reacts to current load, which is too slow for pre-warming workers before a run starts. test-scale pre-scales before dispatch so workers are ready when the run begins.

The KEDA loop (step by step)

sequenceDiagram
    participant User
    participant CP as Control Plane
    participant Brain
    participant Gauge as testscale_desired_workers
    participant Prom as Prometheus
    participant KEDA
    participant K8s as Worker Deployment
    participant Worker

    User->>CP: POST /api/v1/runs
    CP->>Brain: PreScale(target_rps=2000)
    Brain->>Brain: desired = ceil(2000/500) = 4
    Brain->>Gauge: SetDesiredWorkers(4)
    Prom->>Gauge: scrape /metrics every 5s
    KEDA->>Prom: query testscale_desired_workers
    Prom-->>KEDA: value = 4
    KEDA->>K8s: scale worker Deployment → 4 replicas
    K8s-->>Worker: start pods
    Brain->>Prom: poll kube_deployment_status_replicas_available
    Brain-->>CP: 4 workers ready
    CP->>Worker: NATS start signal
    Note over CP,Worker: run executes…
    CP->>Brain: Drain()
    Brain->>Gauge: SetDesiredWorkers(0)
    KEDA->>K8s: scale to minReplicaCount (0)

1. Submit run — orchestrator calls Brain.PreScale().
2. Brain computes — e.g. 4 workers for 2000 RPS.
3. KedaScaler writes — sets testscale_desired_workers gauge on the control plane /metrics endpoint.
4. Prometheus scrapes — ctrl-plane and aggregator targets (Helm configures this).
5. KEDA polls Prometheus — ScaledObject runs query: testscale_desired_workers with threshold: "1".
6. KEDA scales Deployment — with threshold 1, a metric value of 4 means KEDA targets 4 replicas (ceil(metric / threshold)).
7. Brain waits — polls kube_deployment_status_replicas_available until ready count ≥ desired (requires kube-state-metrics in the cluster).
8. Run dispatches — workers register over NATS and load starts.
9. Drain — on completion, gauge → 0; KEDA scales workers to zero after cooldownPeriod (30s in the Helm chart).

Key objects

Object	Role
ScaledObject	Links worker Deployment to Prometheus trigger
testscale_desired_workers	Gauge the control plane sets; KEDA reads
kube_deployment_status_replicas_available	Brain reads to confirm pods are ready

Prerequisites

1. Kubernetes cluster (1.25+)
2. KEDA installed (helm install keda kedacore/keda --namespace keda --create-namespace)
3. kube-state-metrics (for Brain capacity polling)
4. Container images built and pushed (or use imagePullPolicy: Never with local images)

Quick start (kind — local validation)

Your current kubectl context may point at a remote cluster (e.g. AKS). Phase 3 local validation uses kind so you do not need cloud permissions.

# Install kind (macOS)
brew install kind

# One-command bring-up + smoke test
make k8s-verify

# Or step by step
make k8s-up      # kind cluster + KEDA + kube-state-metrics + helm install
make k8s-smoke   # submit ci-smoke.yaml and wait for completion
make k8s-down     # uninstall (DELETE_CLUSTER=1 make k8s-down removes kind cluster)

Scripts live in deploy/kubernetes/scripts/ (up.sh, smoke.sh, down.sh).

Install (manual)

# Build images (from repo root)
docker build -f deploy/docker/Dockerfile.ctrl-plane -t test-scale:local .
docker build -f deploy/docker/Dockerfile.worker -t test-scale:local .
docker build -f deploy/docker/Dockerfile.aggregator -t test-scale:local .

# Install the platform
helm install test-scale deploy/kubernetes/helm/test-scale \
  --set image.repository=test-scale \
  --set image.tag=local \
  --set image.pullPolicy=Never

Configure the control plane

# test-scale.yaml (or env vars)
mode: kubernetes

autoscaler:
  backend: keda          # or k8s — same code path
  per_worker_rps: 500
  prometheus_url: http://test-scale-prometheus:9090
  k8s_namespace: default
  k8s_deployment: test-scale-worker

Environment equivalents:

AUTOSCALER_BACKEND=keda
PROMETHEUS_URL=http://test-scale-prometheus:9090
K8S_DEPLOYMENT=test-scale-worker

Run a test

kubectl port-forward svc/test-scale-ctrl-plane 8080:8080

./test-scale submit -f test-plan.yaml
./test-scale status --run-id <slug> --watch

Verify KEDA is scaling

# Desired count set by control plane
curl -s http://localhost:8080/metrics | grep testscale_desired_workers

# KEDA ScaledObject status
kubectl get scaledobject test-scale-worker -o yaml

# Worker pods
kubectl get pods -l app.kubernetes.io/component=worker -w

What Phase 3 includes vs. Phase 4

Phase 3 (this)	Phase 4 (later)
Helm chart (ctrl-plane, worker, aggregator, NATS, Postgres, Prometheus)	Ingress + cert-manager — see 07-k8s-ingress-tls.md
KEDA ScaledObject	Multi-namespace isolation
KedaScaler backend	Resource quotas
/metrics on control plane	Geo-distributed workers
Managed Postgres connection string support	Cluster-level observability

External access (Ingress + TLS)

For production or kind development without port-forward, see 07-k8s-ingress-tls.md.

Quick start:

make k8s-ingress-up
echo '127.0.0.1 test-scale.local' | sudo tee -a /etc/hosts
open https://test-scale.local:30443

Troubleshooting

Symptom	Likely cause
Workers stay at 0	KEDA not installed, or Prometheus not scraping /metrics
ScaledObject READY=False	KEDA runs in the keda namespace — use the full Prometheus DNS (test-scale-prometheus.test-scale.svc.cluster.local), not a short name
insufficient worker capacity	kube-state-metrics missing; Brain cannot read ready replicas
Workers never drain	Drain() not called — check run completion / orchestrator logs
KEDA shows False activation	testscale_desired_workers is 0 or not in Prometheus

See also 04-deployment-models.md for the full Compose vs Kubernetes comparison.