Interpreting results

test-scale is distributed in architecture (many workers, queue, autoscaler) but deployment mode determines how realistic that distribution is for your question.

Deployment modes vs realism

Mode	Load generation	Best for
Docker Compose	Many containers, one machine	Dev, CI smoke, same-host regression
Kubernetes + KEDA	Pods across cluster nodes	Higher throughput, scale-out generators
Geo-distributed (planned)	Multiple regions	Production-like client mix

On a laptop, “distributed” means parallel containers on shared CPU, memory, and network — not worldwide clients.

What a completed run answers

✓ Did the target meet your SLO at the requested RPS from this generator setup?

✓ Did test-scale orchestrate, scale workers, and report correctly?

✗ Does production handle peak traffic with real geography, CDN, and WAF behavior?

✗ Absolute “we support N users” without matching generator scale and environment.

When results are trustworthy

Use test-scale for relative comparisons:

Pattern	How
Regression over time	Same plan, same target_rps, same deployment mode → compare run slugs
CI smoke	Fixed plan, require error_count: 0 (make docker-smoke)
Release gate	test-scale diff --fail-on-regression vs known-good baseline

Document generator context in plan names or notes: worker count, Compose vs K8s, machine type.

When results mislead

Signal	Likely cause
RPS stuck below target_rps	Generator saturated — add workers, lower per_worker_rps, or move to K8s
Great p95 locally, bad in prod	Single-host path; no edge/cache/routing fidelity
Erratic p95 on a laptop	CPU throttling, Docker Desktop limits, background apps
One green run, no baseline	Noise — rerun and compare

Recommendations

1. Prefer regression over absolutes — treat runs as repeatable experiments, not production certification.
2. Calibrate workers — find sustainable RPS per worker, then let the brain compute count.
3. Escalate deployment — Compose → Kubernetes → (future) multi-region as confidence requirements grow.
4. Find the bottleneck — if worker CPU is maxed, you learned about the load generator, not the API.

Metrics cheat sheet

Metric	Meaning
rps	Achieved aggregate requests/sec
p50_ms / p95_ms / p99_ms	Merged latency percentiles
error_count	Failures + unexpected HTTP statuses
worker_count	Workers that participated

See First run for field definitions on a single run.