Collect measurements close to their source
Prometheus scrapes vLLM’s `/metrics` endpoint every 10 seconds. It also scrapes its own process and a kube-state-metrics Service in the cluster every 15 seconds. The latter is an external dependency: this repository points to a Service in the `monitoring` namespace but does not install kube-state-metrics itself.
The Prometheus configuration and storage settings are declared in the Kubernetes manifest. A 10 GiB claim retains up to 15 days of local samples. This is enough for the demo’s inspection and short comparisons, not a substitute for remote retention, alert delivery, or an operational on-call system.
- job_name: vllm
metrics_path: /metrics
scrape_interval: 10s
static_configs:
- targets: [vllm:8000]
- job_name: kube-state-metrics
scrape_interval: 15s
static_configs:
- targets: [ksm-kube-state-metrics.monitoring.svc.cluster.local:8080]Use a dashboard to ask operational questions
The manifest provisions a Prometheus data source and a vLLM dashboard through ConfigMaps. Its panels cover running and waiting requests, request rate, KV-cache use, queue time, time to first token, end-to-end latency, output-token timing, preemptions, and prefix-cache hits. A second dashboard section shows nodes, pending pods, and the desired and ready replica counts.
These measurements answer different questions. Queue time describes waiting before execution; time to first token describes the delay before output begins; end-to-end latency includes the whole response. Looking at them together helps locate a bottleneck, but none measures whether the generated answer is correct or useful.
Make observability available without another public endpoint
Grafana is a ClusterIP Service, not a public LoadBalancer. An operator reaches it with `kubectl port-forward`, while KEDA queries Prometheus over the cluster network. This keeps the dashboards off the public path in the demo.
Configuration-as-code makes the dashboard repeatable, but both Grafana and Prometheus are single-replica local services. The stack has no alerting or remote-write path in the supplied deployment, and its metrics depend on the cluster’s kube-state-metrics installation being present and reachable.