Why your Kubernetes cluster costs 3x too much

A cluster that costs three times what it should is rarely running three times the workload. It is usually running the right workload, badly sized, on the wrong pricing model, with a layer of things nobody is willing to delete.

I audit these estates for a living. The findings repeat, and they repeat in a predictable order of severity. Here they are, with the reasoning behind each.

1. Requests are a guess that nobody revisited

Someone shipped a service. They needed numbers for resources.requests, so they picked numbers. Those numbers are still there.

This matters more than it sounds, because requests — not actual usage — are what the scheduler packs against. A pod requesting 2 CPU and using 0.2 occupies 2 CPU worth of schedulable space on that node. Ten of those, and you are paying for a node that is 90% idle and 100% full.

Start by finding the gap between requested and used:

# What each pod is actually consuming
kubectl top pods --all-namespaces --sort-by=cpu

kubectl top is a point-in-time reading, so do not resize off one sample. Use a week of Prometheus history and size against a high percentile — not the peak, not the mean:

quantile_over_time(0.95,
  rate(container_cpu_usage_seconds_total{namespace="prod",container!=""}[5m])[7d:5m]
)

P95 over a week gives you a request that absorbs normal spikes without paying for the one pathological Tuesday. On an audit of an enterprise SaaS estate, rightsizing against observed utilization rather than provisioning-time guesses was the single largest line item in a ~40% reduction — and no capacity was removed to get it.

Where this goes wrong: rightsizing memory is not symmetric with CPU. A container over its CPU request gets throttled; a container over its memory limit gets OOM-killed. Be aggressive on CPU requests, conservative on memory limits.

2. Non-production runs at 3am

Development and staging clusters typically serve people in roughly one working day, in roughly one set of time zones. Left alone, they bill for all 168 hours in the week.

A five-day, ten-hour working window is 50 of those 168 hours. Everything else is a nightly charge for an idle environment.

# Scale a non-prod namespace to zero outside working hours
apiVersion: batch/v1
kind: CronJob
metadata:
  name: scale-down-nonprod
  namespace: platform
spec:
  schedule: "0 20 * * 1-5"   # 20:00, weekdays — cluster timezone
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: scaler
          restartPolicy: OnFailure
          containers:
            - name: kubectl
              image: bitnami/kubectl:latest
              command: ["/bin/sh", "-c"]
              args:
                - kubectl scale deploy -l cost.scaledown=allowed --replicas=0 -n staging

Pair it with a scale-up job at the start of the working day. Two caveats before you ship this:

• Do not use --all. Anything running a migration, a queue consumer draining a backlog, or a nightly integration suite will be stopped mid-flight. Select on a label, as above, and let teams opt a workload out by removing it.
• Scaling deployments to zero does not release nodes. The cluster autoscaler has to reclaim them, and it will not evict pods with local storage or restrictive PodDisruptionBudgets. If your node count does not follow your replica count down, that is where to look.

3. Steady baseline load paid at on-demand rates

Every cluster has a floor — the capacity that is running at 4am on a Sunday because it always is. Paying on-demand rates for that floor is paying a premium for flexibility you provably are not using.

Three tiers, mapped to three kinds of workload:

Workload	Pricing model	Why
The floor that never goes away	Reserved / committed use	Predictable by definition. Commit and take the discount.
Normal daytime variation	On-demand	You are buying elasticity, and here you actually use it.
Batch, CI, training, anything restartable	Spot / preemptible	Interruption is an inconvenience, not an outage.

The third row is where the largest untapped savings usually sit, and the reason is cultural rather than technical: the first time a spot node is reclaimed mid-job, someone files an incident, and the team quietly moves everything back to on-demand. Set the expectation before you migrate, not after the first reclaim.

Separate the tiers with node pools and let workloads choose:

spec:
  tolerations:
    - key: "spot"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: "workload-class"   # label your spot pool with this
                operator: In
                values: ["spot"]

4. Untagged resources, which is the one that compounds

The first three are sizing problems. This one is different in kind, and it is why estates regress after a successful optimization.

Without ownership metadata, "is this safe to delete?" has no answer. Nobody deletes anything that might belong to someone else, so nothing gets deleted, and waste accumulates permanently. On an estate of 100+ projects, untagged resources were the root difficulty — not the biggest single number, but the reason the other numbers kept coming back.

The fix is a provisioning requirement, not a cleanup script. Make ownership a condition of creation:

# A variable with no default cannot be omitted
variable "owner_team" {
  type        = string
  description = "Team accountable for this resource's cost."
}

resource "azurerm_resource_group" "app" {
  name     = var.name
  location = var.location

  tags = {
    owner       = var.owner_team
    environment = var.environment
    cost_center = var.cost_center
    managed_by  = "terraform"
  }
}

Enforce it at the policy layer too, so resources created outside Terraform cannot skip it — Azure Policy, AWS SCPs, or an admission controller such as Kyverno for in-cluster objects.

The part that actually determines whether it lasts

An audit is the easy half. Any competent engineer with read access to a billing console can find the waste in a week.

The reason most cost programmes regress within two quarters is that the finding was delivered as a number, and a number is not an owner. What stops the climb is making spend visible and attributable continuously — in the format the finance and delivery teams already use, not a dashboard someone has to remember to open. Adoption beats elegance, every time.

On the estate above, spend is still ~33% below the original baseline while the platform's workload has grown. That durability came from the tagging policy and the cost reporting, not from the rightsizing. The rightsizing produced the drop; the governance is why it stayed down.

If you cut before you can attribute, you will cut again next year.