Measure work at the inference boundary

vLLM exposes counts of requests that are running and waiting. Prometheus scrapes those metrics, and KEDA evaluates their sum through an in-cluster Prometheus address. The `ScaledObject` targets the vLLM Deployment, polls every 15 seconds, keeps at least one replica, allows up to three, and uses a 120-second cooldown.

The configured threshold is two requests. It is a demonstration policy, not a service-level objective: it does not directly say how many seconds a user may wait, nor guarantee that new capacity will be ready before a queue grows.

query: sum(vllm:num_requests_running) + sum(vllm:num_requests_waiting)
threshold: "2"
pollingInterval: 15
cooldownPeriod: 120
minReplicaCount: 1
maxReplicaCount: 3

Scale the proxy by a different signal

Open WebUI has its own HPA: it scales from one to five replicas when average CPU utilization reaches 70%. The manifest also defines a PodDisruptionBudget with `minAvailable: 1`, which constrains voluntary disruptions such as a node drain; it does not make the application highly available during every failure.

The recorded load test found Open WebUI CPU around 4–9%, and it stayed at one replica. The UI mostly forwards requests, so its CPU was not the saturated resource. The vLLM Deployment, by contrast, scaled from one to three replicas. This is why separate controllers and separate metrics matter.

Autoscaling has a response time of its own

A new vLLM pod must schedule, download the model into its private cache, initialize the engine, and pass its health checks. The repository reports roughly two to three minutes per new replica on this cluster. A queue-based trigger can request capacity quickly, but cold-start time means the added capacity may arrive after users have already experienced a long wait.

KEDA does not create compute capacity beyond the cluster’s node budget, and it cannot make a small CPU model behave like a GPU service. Thresholds, cooldowns, node provisioning, and model download strategy must be evaluated together; a good trigger alone is not an end-to-end scaling plan.