GPU scheduling on Kubernetes: a practical guide

A GPU node costs 5-10x what a CPU node costs per hour. Getting GPU scheduling wrong on Kubernetes means paying for idle GPUs, or worse — having GPUs available and no workload scheduled on them while a data scientist waits. This guide covers the setup, the scheduling rules and the cost controls.

The device plugin

Before Kubernetes can schedule GPU workloads, it needs to know which nodes have GPUs and how many. The NVIDIA device plugin registers GPU capacity as a schedulable resource:

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.15.0/deployments/static/nvidia-device-plugin.yml

Once installed, each node with a GPU advertises nvidia.com/gpu as a resource:

kubectl describe nodes | grep nvidia.com/gpu
# nvidia.com/gpu: 4

A pod requests a GPU the same way it requests CPU or memory:

resources:
  limits:
    nvidia.com/gpu: 1

A GPU is not shareable by default — one pod gets the whole device. If your workload underutilises the GPU (inference with a model that fits in a fraction of VRAM), configure time-slicing or MIG (Multi-Instance GPU) on the device plugin side. Time-slicing gives you more scheduling slots; MIG gives you hardware isolation.

Node pools: separating GPU from CPU

Do not put GPUs and CPUs in the same node pool. A GPU node pool should be labelled and tainted so only GPU workloads land there:

# On the GPU node group — Terraform or cloud console
labels:
  workload-class: gpu
  nvidia.com/gpu.present: "true"
taints:
  - key: workload-class
    value: gpu
    effect: NoSchedule

Pods that need GPUs tolerate the taint and request the resource:

spec:
  tolerations:
    - key: workload-class
      value: gpu
      effect: NoSchedule
  containers:
    - resources:
        limits:
          nvidia.com/gpu: 1

A pod that does not request nvidia.com/gpu will not land on a GPU node even if it tolerates the taint, because the default scheduler checks resource availability. But the taint prevents CPU-only pods that accidentally match the node selector from landing there.

Mixed spot and on-demand for cost

GPU instances on spot/preemptible pricing are 60-90% cheaper than on-demand. Training jobs that can tolerate interruption belong on spot. Inference workloads that serve live traffic belong on on-demand or reserved.

Two node pools for GPU:

# Spot GPU pool — for training and batch inference
labels:
  workload-class: gpu-spot
  nvidia.com/gpu.present: "true"
taints:
  - key: workload-class
    value: gpu-spot
    effect: NoSchedule

# On-demand GPU pool — for serving
labels:
  workload-class: gpu-on-demand
  nvidia.com/gpu.present: "true"
taints:
  - key: workload-class
    value: gpu-on-demand
    effect: NoSchedule

A training job tolerates the spot pool and includes checkpointing so it can resume if the node is reclaimed:

spec:
  tolerations:
    - key: workload-class
      value: gpu-spot
      effect: NoSchedule
  containers:
    - env:
        - name: CHECKPOINT_DIR
          value: /checkpoints
      volumeMounts:
        - name: checkpoints
          mountPath: /checkpoints
  volumes:
    - name: checkpoints
      persistentVolumeClaim:
        claimName: training-checkpoints

The checkpoint volume survives node reclamation. The training script writes checkpoints periodically; the job that restarts picks up the latest checkpoint and resumes.

GPU monitoring

GPUs have metrics that CPUs do not: VRAM usage, GPU utilisation, temperature and power draw. The NVIDIA DCGM exporter exposes these to Prometheus:

helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \
  --namespace monitoring \
  --set serviceMonitor.enabled=true

The metrics that matter:

# GPU utilisation — is the GPU doing work or sitting idle?
DCGM_FI_DEV_GPU_UTIL

# Framebuffer usage — is VRAM full? If so, the workload is about to OOM
DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL

# GPU temperature — thermal throttling starts around 85°C on most cards
DCGM_FI_DEV_GPU_TEMP

An idle GPU at 100% VRAM is a GPU with a workload that crashed and leaked memory. An idle GPU at 0% utilisation with a pod scheduled on it is a pod that started and did nothing — find it and terminate it.

Cost controls that actually work

• Set GPU resource quotas per namespace. A namespace with no quota can consume every GPU in the cluster. A single misconfigured job takes down every other GPU workload:

  apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: gpu-quota
    namespace: ml-team
  spec:
    hard:
      nvidia.com/gpu: "4"

• Schedule GPU nodes to shut down outside working hours — the same pattern as non-production CPU workloads. A GPU that is idle from 8pm to 8am bills for 12 hours of nothing. A CronJob that scales the node group to zero outside working hours pays for zero of those hours.

• Tag GPU resources by project and owner. A GPU instance with no owner tag is a GPU instance nobody will turn off. The tagging policy that works for CPU cost works for GPU cost — it just matters 5-10x more.

The GPU infrastructure reality

The hard part of GPU scheduling is not Kubernetes. It is the waiting: GPU instances are capacity-constrained in every cloud region, and spot GPU capacity is scarcer still. A training job that needs 8 A100s may sit in the scheduler queue for hours waiting for capacity.

Mitigations, in order of effectiveness:

1. Request GPU quota increases before you need them. Cloud providers approve GPU quota on a timeline, not instantly.
2. Run training on spot and fall back to on-demand when spot is unavailable. The job that waits an hour for a spot node is still cheaper than the job that runs immediately on on-demand — and the difference is not small.
3. Use smaller GPU types if your model fits. An A10 or L4 is more available and cheaper than an A100. Only pay for the A100 if the model genuinely needs it.

---

Running GPU workloads or building an ML platform? I take on AI/ML infrastructure and Kubernetes platform builds as freelance and contract work — independently, for teams in the US, EU and APAC.