Scheduling GPUs on Kubernetes

Setup#

No installs needed for the simulation — this uses only Python's standard library.

Simulating the core scheduling decision#

from dataclasses import dataclass

@dataclass
class Node:
    name: str
    gpus_total: int
    gpus_used: int = 0

    @property
    def gpus_free(self):
        return self.gpus_total - self.gpus_used

@dataclass
class Pod:
    name: str
    gpus_requested: int

def schedule(nodes, pods):
    """First-fit: place each job on the first machine with enough free GPUs.
    This mirrors the core idea of the Kubernetes scheduler filtering nodes
    by a resource request, simplified to one resource type."""
    placements, pending = {}, []
    for pod in pods:
        placed = False
        for node in nodes:
            if node.gpus_free >= pod.gpus_requested:
                node.gpus_used += pod.gpus_requested
                placements[pod.name] = node.name
                placed = True
                break
        if not placed:
            pending.append(pod.name)
    return placements, pending

nodes = [Node("gpu-node-1", gpus_total=4), Node("gpu-node-2", gpus_total=2)]
pods = [
    Pod("training-job-a", gpus_requested=2),
    Pod("inference-svc-b", gpus_requested=1),
    Pod("training-job-c", gpus_requested=3),
    Pod("inference-svc-d", gpus_requested=1),
    Pod("training-job-e", gpus_requested=2),
]

placements, pending = schedule(nodes, pods)

print("scheduling result:")
for pod_name, node_name in placements.items():
    print(f"  {pod_name:16} -> {node_name}")
if pending:
    print("pending (no node had enough free GPUs):")
    for p in pending:
        print(f"  {p}")

print("\nnode state after scheduling:")
for node in nodes:
    print(f"  {node.name}: {node.gpus_used}/{node.gpus_total} GPUs used")

scheduling result:
  training-job-a   -> gpu-node-1
  inference-svc-b  -> gpu-node-1
  inference-svc-d  -> gpu-node-1
  training-job-e   -> gpu-node-2
pending (no node had enough free GPUs):
  training-job-c

node state after scheduling:
  gpu-node-1: 4/4 GPUs used
  gpu-node-2: 2/2 GPUs used

This is real, deterministic Python — rerun it and you get exactly this result every time. training-job-c asked for 3 GPUs. By the time it was considered, node-1 had only 1 free and node-2 only had 2 total. Six free GPUs existed across the whole cluster at various points. Yet this one job could not be placed — a genuine and common scheduling failure mode, not a bug in this script.

A real Kubernetes GPU request, for reference#

This is correct, real Kubernetes YAML — the configuration a real job would actually use to ask for a GPU. There is no output block for it. Running it needs a real cluster with the NVIDIA device plugin installed, which cannot exist inside a single script.

apiVersion: v1
kind: Pod
metadata:
  name: training-job-a
spec:
  containers:
    - name: trainer
      image: my-training-image:latest
      resources:
        limits:
          nvidia.com/gpu: 2   # this pod is asking for exactly 2 GPUs

nvidia.com/gpu is a special resource name the NVIDIA device plugin registers with Kubernetes. The scheduler can then count and allocate whole GPUs, the same way it already counts CPU and memory.

Line-by-line walkthrough#

first-fit, checking nodes in order. Real Kubernetes uses a more sophisticated scoring system by default. But the underlying filter — "does this node have enough of what was asked for?" — is the same idea this simulation makes concrete.

training-job-c going pending. This is the single most important thing to notice in the whole demo. Pending is not an error. It means the scheduler correctly found nowhere the job currently fits, and is waiting for room, exactly as designed.

GPUs are requested as a whole number, never a fraction, in standard Kubernetes. nvidia.com/gpu: 2 means exactly two entire GPUs, allocated to this one pod alone. Splitting a single GPU between pods needs one of the sharing mechanisms from sharing one GPU between models, not a fractional request here.

Common mistakes#

Requesting more GPUs than any single machine in the cluster has. A request for 8 GPUs will sit pending forever on a cluster where the biggest machine only has 4. That holds even if the cluster's total GPU count across many machines is much higher than 8 — Kubernetes schedules onto one node, not split across several, for a single pod.

Assuming "pending" means something is broken. As the simulation shows, a job can be perfectly correctly configured and still wait, because of real, legitimate resource pressure. Check node capacity before assuming a configuration problem.

Fragmentation from placing large jobs after small ones. In the simulation, placing the 3-GPU job earlier would have let it schedule successfully — before node-1 filled up with smaller ones. Real schedulers offer bin-packing strategies specifically to reduce this kind of waste; see the researcher section.

Forgetting the device plugin. A cluster with physical GPUs will not report nvidia.com/gpu as a schedulable resource at all. Not until the NVIDIA device plugin, or an equivalent, is installed and running. Without it, GPU requests are invalid outright, not only unscheduled.

Try it yourself#

Reorder the pods list so training-job-c (3 GPUs) is scheduled first, before the smaller jobs. Rerun the simulation and see whether every job gets placed this time.

What to learn next#

• Docker for ML — packaging the workload that ends up inside a scheduled pod.
• Running on spot and preemptible GPUs — what happens when a scheduled GPU can be taken away mid-job.
• Sharing one GPU between models — the fractional-GPU sharing this lesson's whole-number scheduling model does not cover on its own.