Measure the full response, while watching the cluster
`loadtest/run.sh` executes a baseline, then phases with 4, 12, and 24 concurrent clients, followed by a cooldown. During the run it samples Prometheus and Kubernetes state every 15 seconds. The standard-library generator repeatedly posts a short prompt with a small output cap and records status codes and full-response latency; it does not measure answer quality.
The README reports that the 24-client phase completed 394 of 394 requests at about 1.51 requests per second, with an end-to-end p95 of 23.3 seconds. KEDA scaled vLLM from one to three replicas, while Open WebUI stayed at one. At lighter phases, some requests were disconnected near the load balancer’s roughly 50-second idle timeout.
"rps": round(len(results) / wall, 2) if wall else 0,
"latency_s": {
"p50": round(pct(0.5), 2),
"p95": round(pct(0.95), 2),
"p99": round(pct(0.99), 2),
}Read the result in the context of its workload
The test used a tiny prompt and a maximum of 16 generated tokens on a three-node CPU cluster. Its request rate is not token throughput, and its p95 is not a time-to-first-token metric. It does show that queue-based scaling happened and that short requests could complete at high concurrency; it also shows that added model capacity arrives slowly and tail latency remains substantial.
The checked-in harness contains demo sign-in behavior, so anyone adapting it should replace that with a dedicated test identity and run only against a cluster they control. The generated run directory is ignored by Git, even though the README summarizes one run. The scripts are available for inspection, but the raw measurement artifacts are not part of the public revision.
Keep the useful architecture, replace the demo assumptions
The project’s production path keeps the stable API boundary and queue-based scaling, but changes the serving image and hardware to GPUs, shares or remotely sources model weights, moves Open WebUI files to object storage, and puts TLS and OIDC at an ingress boundary. It also calls for pinned image digests, least-privilege access, managed secrets, and remote metrics with alerting.
The main lesson is not that this small CPU stack is ready for production. It is that a working inference service depends on the surrounding system: model memory, storage semantics, queue signals, network timeouts, identity, and measurements. The complete implementation and setup notes are in the LLM Stack repository. Questions, corrections, and alternative designs are welcome.