Collect measurements close to their source

Prometheus scrapes vLLM’s `/metrics` endpoint every 10 seconds. It also scrapes its own process and a kube-state-metrics Service in the cluster every 15 seconds. The latter is an external dependency: this repository points to a Service in the `monitoring` namespace but does not install kube-state-metrics itself.

The Prometheus configuration and storage settings are declared in the Kubernetes manifest. A 10 GiB claim retains up to 15 days of local samples. This is enough for the demo’s inspection and short comparisons, not a substitute for remote retention, alert delivery, or an operational on-call system.

- job_name: vllm
  metrics_path: /metrics
  scrape_interval: 10s
  static_configs:
    - targets: [vllm:8000]

- job_name: kube-state-metrics
  scrape_interval: 15s
  static_configs:
    - targets: [ksm-kube-state-metrics.monitoring.svc.cluster.local:8080]

Use a dashboard to ask operational questions

The manifest provisions a Prometheus data source and a vLLM dashboard through ConfigMaps. Its panels cover running and waiting requests, request rate, KV-cache use, queue time, time to first token, end-to-end latency, output-token timing, preemptions, and prefix-cache hits. A second dashboard section shows nodes, pending pods, and the desired and ready replica counts.

These measurements answer different questions. Queue time describes waiting before execution; time to first token describes the delay before output begins; end-to-end latency includes the whole response. Looking at them together helps locate a bottleneck, but none measures whether the generated answer is correct or useful.

Make observability available without another public endpoint

Grafana is a ClusterIP Service, not a public LoadBalancer. An operator reaches it with `kubectl port-forward`, while KEDA queries Prometheus over the cluster network. This keeps the dashboards off the public path in the demo.

Configuration-as-code makes the dashboard repeatable, but both Grafana and Prometheus are single-replica local services. The stack has no alerting or remote-write path in the supplied deployment, and its metrics depend on the cluster’s kube-state-metrics installation being present and reachable.