Begin with the request path
A user opens Open WebUI through the cluster’s public LoadBalancer. Open WebUI then calls an internal, OpenAI-compatible endpoint exposed by vLLM. The browser does not need to know where the model pod runs: Kubernetes Service DNS gives the interface a stable name even when individual pods are replaced.
Other components support that request rather than generate its answer. PostgreSQL stores Open WebUI application data and provides the configured pgvector integration. Redis coordinates WebSocket connections and acts as a cache. Prometheus collects inference and cluster metrics; Grafana reads those metrics; KEDA queries Prometheus to scale vLLM. A separate HPA watches CPU use by Open WebUI.
Browser
-> public Service: openwebui
-> Open WebUI
-> vLLM Service: vllm:8000/v1
-> PostgreSQL and Redis
vLLM /metrics -> Prometheus -> Grafana
-> KEDA -> vLLM replicasTreat the hardware limit as a design input
The demonstration uses three small CPU nodes rather than a GPU server. That keeps the experiment inexpensive and makes scheduling, memory limits, storage, and scaling visible. The same topology can teach how Kubernetes components communicate, but its capacity is intentionally modest: model inference is slow, and the model itself is small.
A single virtual machine would have fewer moving parts. A GPU cluster would serve larger models more efficiently. This project chooses Kubernetes on CPU because the learning objective is to operate a complete stack under constraint—not to claim the best latency or answer quality for a production assistant.
Keep the boundary between demo and service clear
Only Open WebUI is exposed with a LoadBalancer; the model API, database, Redis, Prometheus, and Grafana use internal Services. That reduces the number of public entry points, but it does not make the deployment production-ready. The public endpoint is plain HTTP, and the repository’s production notes call for TLS, stronger identity management, and managed secrets.
The next note zooms into the most constrained component: how vLLM is configured to fit model weights, context, shared memory, and concurrent requests into a small CPU pod.