Deployment models

1. Design principle: one codebase, two runtimes

All components (control plane, workers, autoscaler, aggregator, metrics) are the same Go binaries and the same container images in both modes. What changes is how they are orchestrated, discovered, and scaled.

This is not a “dev vs prod” split — both modes are production-capable. The right mode is the one that fits the team’s infrastructure maturity.

flowchart TB
    subgraph Apps["Application containers (identical)"]
        CP[Ctrl Plane]
        WK[Worker]
        AG[Aggr.]
        GF[Grafana]
    end

    subgraph Orch["Orchestration layer (swappable)"]
        DC[Docker Compose<br/>local / single machine]
        K8S[Kubernetes<br/>cloud cluster]
    end

    subgraph Infra["Infrastructure"]
        subgraph Local["Local"]
            PGL[Postgres container]
            NATSL[NATS container]
            PROML[Prometheus container]
        end
        subgraph Cloud["Cloud"]
            PGC[Managed Postgres<br/>RDS / Cloud SQL]
            NATSC[Managed NATS / self-hosted]
            PROMC[Prometheus Operator / Grafana Cloud]
        end
    end

    DC --> Local
    K8S --> Cloud

---

2. What is shared vs. what differs

Shared (single code path)

Concern	How
Control plane binary	Same Go code, same Dockerfile
Worker binary	Same Go code, same Dockerfile
Aggregator binary	Same Go code, same Dockerfile
Postgres schema	Versioned SQL migrations in both modes
NATS subjects & protocol	Same subjects, same message format
Test plan API contract	Identical REST API
Metrics protocol	Same OTLP/OpenTelemetry format
Dashboard queries	Same PromQL / Grafana dashboards

Differs (config-driven, not code-forked)

Concern	Docker Compose	Kubernetes
Orchestration	docker compose up --scale worker=N	Deployment + KEDA ScaledObject
Service discovery	Docker DNS (worker:8080)	K8s DNS (worker.default.svc.cluster.local)
Autoscaler backend	compose.DockerScaler — calls Docker API	kube.KedaScaler — updates KEDA target metric
TLS termination	None (localhost)	cert-manager + Ingress
Secret management	.env file	K8s Secrets
Persistent volumes	Docker named volumes	PVC / cloud block storage
Ingress	localhost:{port}	Ingress controller
Log aggregation	Docker logs / stdout	Structured logs to stdout → cluster-level collector

Both modes respect the same test-scale.yaml or environment variables. The discovery mechanism and autoscaler backend are selected via a top-level config key:

mode: local | kubernetes

---

3. Docker Compose mode (local / single-machine)

When to use it

• Developer workstation testing (build → run → validate → tear down)
• CI/CD smoke tests (PR pipeline: Compose up → 30s load test → Compose down)
• Small-team use where one beefy machine handles all load generation
• Quick ad-hoc tests — no cluster provisioning overhead

Architecture

flowchart TB
    subgraph Host["Docker Host"]
        CP[Ctrl Plane<br/>:8080]
        AG[Aggregator<br/>:4317 OTLP / :9464 Prometheus]
        GF[Grafana<br/>:3000]
        NATS[NATS<br/>:4222]
        PROM[Prometheus<br/>:9090]
        W1[Worker 1]
        W2[Worker 2]
        WN[Worker N]
        PG[Postgres<br/>:5432]
    end

    CP <--> NATS
    AG --> PROM
    PROM --> GF
    W1 --> PROM
    W2 --> PROM
    WN --> PROM
    W1 <--> NATS
    W2 <--> NATS
    WN <--> NATS
    W1 --> AG
    W2 --> AG
    WN --> AG
    AG --> PROM
    CP --> PG
    AG --> PG

Scaling model

• Worker count is set by the test plan’s worker_count field (Phase 1–2).
• Phase 3+: DockerScaler calls docker compose scale worker=N through the Docker socket API, driven by the same autoscaler logic the K8s path uses.
• Scale down: docker compose scale worker=0 after a test completes, stopping all worker containers. Infrastructure containers (Postgres, NATS, Prometheus) stay running.

Startup (developer laptop)

docker compose up -d                          # infra + 0 workers
./test-scale submit -f plan.yaml              # submits test, triggers scaling
./test-scale watch --run-id abc123            # live dashboard URL
docker compose down                           # tear everything down

---

4. Kubernetes mode (cloud cluster)

When to use it

• Production load testing at scale (thousands of RPS)
• Cross-team platform (multiple engineers running tests concurrently)
• Multi-node worker distribution (simulate geo-distributed traffic)
• Persistent deployment — platform stays up, scales workers on demand

Architecture

flowchart TB
    subgraph K8S["Kubernetes Cluster"]
        CPD["Deployment: ctrl-plane<br/>(2 replicas)"]
        AGD["Deployment: aggregator<br/>(1 replica)"]
        NATSS["StatefulSet: NATS<br/>(3 replicas)"]
        PROMD["Deployment: Prometheus Operator"]
        KEDA["KEDA ScaledObject: worker<br/>Min: 0, Max: 500<br/>Trigger: Prometheus metric"]
        WKD["Deployment: worker<br/>(scales 0 ↔ N)"]
        ING["Ingress (nginx)"]
        CERT["cert-manager"]
    end

    subgraph External["External (managed or self-hosted)"]
        PGRDS["Postgres RDS<br/>(or cloud SQL)"]
        PROMC["Prometheus / Grafana Cloud<br/>(or VictoriaMetrics)"]
    end

    KEDA --> WKD
    CPD --> PGRDS
    AGD --> PGRDS

Managed services (initial cloud setup)

Service	Managed option	Why managed first
Postgres	RDS / Cloud SQL / Flexible Server	Backups, patching, HA — not load-test-specific
NATS	Self-hosted via Helm (3-replica StatefulSet)	No major cloud provider has a managed NATS yet; it’s lightweight
Prometheus	Grafana Cloud / self-hosted Operator	Managed = no retention worries; Operator = zero-cost start
Grafana	Grafana Cloud / self-hosted	Same tradeoff as Prometheus

Teams that later want full control can swap RDS → self-hosted Postgres in-cluster without changing any application code — just the connection string.

Scaling model

• Autoscaler writes a desired replica count to a KEDA-compatible metric (e.g., a Prometheus gauge testscale_desired_workers for the active run).
• KEDA ScaledObject reads that metric and scales the worker Deployment.
• Between runs: desired_workers = 0 → worker pool scales to zero.
• KEDA cooldown: immediate scale-down after a run completes (no 5-minute wait).

Example KEDA ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-scaler
spec:
  minReplicaCount: 0
  maxReplicaCount: 500
  cooldownPeriod: 30
  scaleTargetRef:
    name: worker-deployment
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: testscale_desired_workers
      query: testscale_desired_workers{run_id="active"}
      threshold: "1"

---

5. Autoscaler abstraction

Both modes implement the same Scaler interface:

type Scaler interface {
    // ScaleTo sets the worker pool to exactly n replicas.
    ScaleTo(ctx context.Context, n int) error

    // CurrentCount returns the current number of healthy worker replicas.
    CurrentCount(ctx context.Context) (int, error)

    // Drain scales to zero immediately (run complete or aborted).
    Drain(ctx context.Context) error
}

Implementation	How it works
compose.DockerScaler	Calls Docker API POST /services/{id}/update or docker compose scale
kube.KedaScaler	Updates the Prometheus metric that KEDA reads; KEDA does the actual scaling

The autoscaler brain (deciding what the count should be) is shared. Only the actuator (applying the count) differs.

---

6. Progressive adoption path

flowchart TD
    P0["Phase 0–2 ✅<br/>Docker Compose<br/>(develop, CI, small teams)"]
    P3["Phase 3 ✅<br/>Kubernetes + KEDA<br/>alongside Compose"]
    P3A["Helm chart ✅"]
    P3B["KEDA ScaledObject ✅"]
    P3C["kind smoke test ✅"]
    P3D["cert-manager + Ingress ✅"]
    P3E["Managed Postgres ⏳"]
    P4["Phase 4 🔄<br/>Production hardening"]
    P4A["Verification, compare UI ✅"]
    P4B["CLI watch + open ✅"]
    P4C["Worker mid-run replacement ✅"]
    P4D["Multi-namespace / quotas ⏳"]
    P4E["Geo-distributed workers ⏳"]

    P0 --> P3
    P3 --> P3A
    P3 --> P3B
    P3 --> P3C
    P3 --> P3D
    P3 --> P3E
    P3 --> P4
    P4 --> P4A
    P4 --> P4B
    P4 --> P4C
    P4 --> P4D
    P4 --> P4E

Phase 3 is implemented. Local validation: make k8s-verify (see 05-phase3-kubernetes.md).

At no point is the Compose mode deprecated. Both modes are maintained as first-class citizens because they serve different use cases:

Compose mode	K8s mode
Quick local test	Sustained production load
CI smoke test	Cross-team platform
Zero cost (existing hardware)	Scales to hundreds of workers
No cluster to manage	HA, multi-node

---

7. Configuration surface

# test-scale.yaml — same file works in both modes

mode: local            # local | kubernetes

postgres:
  host: localhost      # or rds-endpoint.aws.com
  port: 5432
  database: testscale

nats:
  url: nats://localhost:4222

autoscaler:
  backend: docker      # docker | keda
  per_worker_rps: 500  # benchmarked capacity of one worker

metrics:
  otel_endpoint: http://aggregator:4317

Environment variable overrides exist for secrets (PG_PASSWORD, etc.) so the config file can be committed to version control.

---

8. CI/CD integration

GitHub Actions (.github/workflows/ci.yml) runs on every push/PR to main:

Job	What it does
test	go test -race with NATS service
build	go build ./...
smoke	Docker Compose up → ./bin/test-scale run → assert run completes

Local equivalent:

./setup.sh
make docker-smoke

Example workflow snippet (see repo for full version):

jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: "1.26.5"
      - run: cp .env.example .env && docker compose -f deploy/docker/docker-compose.yml up -d --build
      - run: deploy/docker/scripts/smoke.sh
      - run: docker compose -f deploy/docker/docker-compose.yml down -v
        if: always()