HPA, VPA and cluster-autoscaler, together

Horizontal Pod Autoscaler is table stakes. But HPA only knows what it observes — CPU, memory, or a custom metric — and it has no opinion about whether a pod was the right size to begin with. That is a different question, answered by a different controller, and the two can be made to work against each other.

Three decisions at three speeds

Kubernetes scaling is not one feature. It is three independent control loops:

Layer	Acts in	Changes	Question it answers
HPA	Seconds	Replica count	"Do we need more copies?"
VPA	Minutes to hours	Requests and limits	"Was each copy the right size?"
cluster-autoscaler	Minutes	Node count	"Is there anywhere to put them?"

They only compose if each has a distinct input. Give two of them the same input and you get oscillation.

The conflict nobody warns you about

Do not run HPA and VPA on the same metric for the same workload.

HPA on CPU adds replicas when CPU utilization — measured as a percentage of the request — rises. VPA in Auto mode raises the CPU request when usage is high. Raising the request lowers the utilization percentage. HPA now sees a workload comfortably under target and scales replicas back down. Load has not changed; the denominator did.

The result is a workload that thrashes, with neither controller wrong on its own terms.

Three safe arrangements:

1. HPA on a custom or external metric, VPA on resources. HPA scales on requests-per-second or queue depth; VPA owns sizing. No shared input, no fight.
2. HPA on CPU, VPA in Off mode. VPA still computes and publishes recommendations, and you apply them deliberately. Slower, and considerably easier to reason about.
3. VPA in Auto, no HPA. Correct for workloads that scale vertically — a single-writer database, a queue consumer whose concurrency is fixed.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"        # recommend only; nothing is mutated
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        controlledResources: ["cpu", "memory"]

Read what it would have done:

kubectl describe vpa api | grep -A6 'Recommendation'

Start every workload here. Auto on a legacy service with no startup probes will evict pods to resize them, and a pod that takes 90 seconds to warm up does not enjoy being restarted for a 50Mi adjustment.

HPA: the target is a ratio, not a ceiling

The most common HPA mistake is treating averageUtilization: 50 as "scale at 50% CPU". It is a steady-state target — HPA adds replicas until average utilization across the fleet returns to 50%.

Which means the number depends entirely on how fast your load moves. A workload that jumps from 10% to 80% in seconds needs headroom, so a 50% target is right. A workload that climbs gradually reaches equilibrium fine at 70%, and a 50% target simply runs 40% more pods than it needs, permanently.

Stabilization windows matter more than the target:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 40
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0     # react immediately
      policies:
        - type: Percent
          value: 100                     # at most double per interval
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300    # wait 5 min before shrinking
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60

Asymmetry is the point. Scaling up late costs you an outage; scaling down early costs you a re-scale two minutes later. Be quick to add and slow to remove.

cluster-autoscaler: why nodes do not go away

Scale-up is the easy direction — pods go Pending, the autoscaler adds a node. Scale-down is where the surprises live, and a node that will not drain is nearly always one of these:

• A PodDisruptionBudget that cannot be satisfied. minAvailable: 100%, or a single-replica deployment with minAvailable: 1, means no pod may ever be evicted. The node stays forever.
• Pods with local storage. emptyDir blocks eviction by default, because the data cannot be moved.
• Pods with no controller. A bare pod has nothing to recreate it, so it is not evicted.
• kube-system pods without a PDB. Common on self-managed clusters.

Ask the autoscaler directly rather than inferring:

kubectl -n kube-system logs deploy/cluster-autoscaler | grep -i 'scale.down\|unremovable'

It names the node and the blocking pod.

The other half is bin packing. Ten nodes at 40% utilization cannot be consolidated if every workload demands anti-affinity across nodes. Scheduling constraints and node reclamation are the same problem viewed from two directions, and topology spread constraints are usually a better tool than hard anti-affinity:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway    # a preference, not a wall
    labelSelector:
      matchLabels:
        app: api

ScheduleAnyway keeps spread as a goal without making it a reason a node can never empty.

What "smart scaling" is not

It is not maxReplicas: 1000 and hope. It is not updateMode: Auto on a legacy monolith with no startup probes. It is not ignoring the bill because uptime is fine.

It is each layer having a distinct input and a distinct speed: pods request what they actually use, HPA scales on a signal a user would notice, and the cluster grows only when pods genuinely cannot fit. Get that arrangement right and the hourly cost graph goes flat and boring — which is the best news it can carry.

The cost side of this — requests set from guesswork, non-production billing overnight, baseline load at on-demand rates — is a separate post.