Setting up Prometheus and Grafana on Kubernetes: a practical guide

Observability is the difference between knowing something is broken and finding out from a user. On Kubernetes, the toolchain that answers this for most teams is Prometheus, Grafana and Alertmanager — three projects from the CNCF ecosystem that together give you metrics collection, dashboards and alerting.

This guide walks through setting them up, deciding what to monitor and configuring the alerts that actually matter.

The stack

Prometheus scrapes metrics from your workloads and the cluster itself. Grafana renders them as dashboards. Alertmanager sends notifications when something crosses a threshold.

Install them via the kube-prometheus-stack Helm chart, which bundles all three plus a set of pre-built recording rules and dashboards:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set prometheus.prometheusSpec.retention=15d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi

The retention and storage values are the minimum, not the ideal. 15 days of metrics at 100 Gi gives you two weeks of history for trend analysis and one week of margin.

What to monitor: the three pillars

A monitoring setup that collects everything and alerts on nothing is as useless as one that collects nothing. These are the three pillars every team needs before going to production.

1. The RED method (per service)

Request rate, error rate and duration. For every service behind an HTTP endpoint, these three metrics tell you whether it is working and whether it is fast:

# Request rate
rate(http_requests_total{service="api"}[5m])

# Error rate (5xx as a fraction of total)
rate(http_requests_total{service="api",status=~"5.."}[5m])
/
rate(http_requests_total{service="api"}[5m])

# 99th percentile latency
histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket{service="api"}[5m])
)

The 99th percentile is the number that matters for user experience. The average hides the slow requests that only a few users see, and those are the ones who leave.

2. Cluster health

Node conditions, pod restarts and resource pressure are the cluster-level signals. If a node goes NotReady, you want to know before a workload is evicted:

# Node condition — fires when any node is NotReady for more than 5 minutes
kube_node_status_condition{
  condition="Ready",
  status="true"
} == 0

Pod restarts are a leading indicator of a problem a user has not reported yet. A pod that restarts twice an hour is a pod that is crashing on a timer, and the crash loop backoff is hiding it:

# High restart rate over the last hour
rate(kube_pod_container_status_restarts_total[1h]) > 0

3. Resource saturation

CPU and memory are the obvious ones, but disk and inodes are the ones that fail silently:

# Node filesystem filling up — alert before the node hits 85%
100 - (
  node_filesystem_avail_bytes{mountpoint="/"}
  /
  node_filesystem_size_bytes{mountpoint="/"}
  * 100
) > 85

# Node memory pressure — when the kernel starts killing processes
kube_node_status_condition{
  condition="MemoryPressure",
  status="true"
} == 1

A node that runs out of disk or inodes fails in ways that CPU or memory exhaustion does not: the kubelet stops accepting new pods, existing pods crash when they try to write logs, and the cluster appears healthy from the outside while the inside is dead.

Alerting: what to wake someone up for

The difference between a page and a ticket is whether a human has to act now. Paging on high CPU is a common mistake — high CPU is a symptom, not an incident. Page on the user-facing symptom, and let the dashboard show the cause.

These five alerts cover the ground:

Alert	Expression	Why
Service error rate > 1%	RED error rate over 5m window	Users are seeing errors right now
Pod crash looping	Restart count increase rate	A deployment is broken and the fix is not self-healing
Node not ready	Node condition false for 5m	The scheduler cannot place workloads on this node
PersistentVolume filling	PV usage > 85% and no PVC resize possible	A stateful workload is about to run out of disk
Certificate expiring	cert-manager certmanager_certificate_expiration_timestamp_seconds	TLS will break silently at the expiry time

The fifth alert is the one nobody writes until their first certificate expiry outage. Set it up now.

Three dashboards every team needs

1. Service overview: RED metrics per service, one row per service, sorted by request rate. The first screen you look at during an incident.

2. Resource usage per namespace: CPU, memory, disk and network per namespace. The dashboard you show when someone asks "do we need a bigger cluster" and the answer is "your namespace is using 80% of the cluster's CPU and you asked for 200% more this month."

3. Deployment health: Pod status, restart count and readiness per deployment. The reconciliation view for "did the last deploy work."

Observability without ownership is expensive

A monitoring stack that ships without an owner decays. Dashboards break when metrics are renamed. Alerts fire and nobody responds, so someone silences them. Within six months the stack is running, consuming resources and producing nothing useful.

The smallest team that can own this is one person who reviews the alerts weekly and the dashboards monthly. If your team cannot commit that one person, start with managed Prometheus (AWS Managed Prometheus, Azure Monitor Managed Prometheus, Grafana Cloud) and accept the per-sample cost in exchange for the operational burden you are not carrying.

---

Setting this up or inheriting a Prometheus stack that has gone silent? I take on Kubernetes and GitOps platform builds and fractional DevOps retainers as freelance and contract work — independently, for teams in the US, EU and APAC.