Use an API contract to separate the UI from inference
Open WebUI is configured with an internal base URL at vLLM’s `/v1` endpoint. It sends chat requests through the Kubernetes Service named `vllm`; the Service forwards them to port 8000 on a ready model pod. Replacing the backend therefore does not require the browser to address a pod or understand the serving runtime, as long as the replacement preserves the expected API.
The selected model is Qwen2.5-0.5B-Instruct, served with the CPU image. It is small enough for this cluster, but model size is also a capability tradeoff: success in loading and answering a short prompt is not evidence that the model is suitable for demanding reasoning or production quality.
args:
- serve
- Qwen/Qwen2.5-0.5B-Instruct
- --dtype
- bfloat16
- --max-model-len
- "8192"
- --max-num-seqs
- "2"
- --kv-cache-memory-bytes
- "536870912"Budget memory across weights, context, and shared memory
The pod requests 4 GiB of memory and is limited to 6 GiB. Bfloat16 reduces the model-weight footprint compared with float32; an explicit 512 MiB KV-cache budget limits memory reserved for active context; and the maximum model length is 8,192 tokens. The number of active sequences is capped at two per replica. These settings make the workload schedulable, but they also constrain how many long conversations can be processed at once.
A memory-backed `emptyDir` supplies up to 1 GiB at `/dev/shm`, which vLLM needs for shared-memory operations. Another `emptyDir` stores the Hugging Face model cache for that pod. It is ephemeral and local to the replica, so a new replica must download the model before it can become ready. Startup probes allow time for loading, but do not remove the cold-start delay.
Account for the interface’s request shape
Open WebUI may include tool specifications even for a short message, and it can request automatic tool choice. The manifest therefore raises the context limit and enables vLLM’s Hermes tool-call parser. The setting `VLLM_PORT` is also made explicit: Kubernetes normally injects environment variables for Services into pods, and a variable derived from the Service name collided with vLLM’s port setting.
Finally, `OMP_NUM_THREADS=2` limits CPU threading. It was chosen after slow prefill caused requests to approach the load balancer’s roughly 50-second idle cutoff. This is a pragmatic mitigation, not a latency guarantee; longer prompts can still queue or exceed that network timeout.