Hey {{first name | there}}. This past weekend I caught up on a lot of reading about inference engines, and more specifically the stages involved in inference.
Somewhere in that rabbit hole, I landed on the Kubernetes 1.37 write-up about workload-aware scheduling, and specifically gang scheduling.
As the title hinted, my first reaction was, “guess we are all in a gang now.” Then I sat with it, because this is the scheduling problem I keep hitting once GPU workers enter the chat.
In today's technical notes:
WTH is gang scheduling
Why partial placement is the actual bill
Why I am willing to try it
📰Technical notes: Hold on, a gang?
Although I am writing about this now, gang scheduling is not new. Volcano, Kueue, and the Kubeflow Training Operator have all done some version of all-or-nothing for years. The KEP even admits it has been implemented outside kube-scheduler at least four times.
What is new is that kube-scheduler itself grew a native version in 1.37. Workload and PodGroup APIs are at scheduling.k8s.io/v1beta1. The feature gate is GenericWorkload. It is beta, and it is still disabled by default.
To quote directly from KEP-4671:
Parallel applications can require communication between every pod in order to begin execution, and then ongoing communication between all pods (such as barrier or all-reduce operations) in order to make progress. Starting all pods as close to the same time is necessary to run these workloads. Otherwise, either expensive compute resources are idle, or the application may fail due to an application-level communication timeout.
That is the whole pitch. Not a fun name. A rule about when the scheduler is allowed to bind.
So….. what is the real problem?
Default Kubernetes schedules Pods one at a time. That is the correct default for a Deployment. It is the wrong default for eight GPU workers that cannot start work until every rank is up.
You create eight Pods. Five find nodes. Three sit in Pending. The five are Running, holding GPUs, waiting on an all-reduce that will never start. The three are waiting on GPUs that those five are already occupying.
Nobody is making progress. You are still paying for five accelerators.
Sadly, this is just scheduler doing exactly what it was asked: place whatever fits. Gang scheduling is the opt-in that says do not place any of them unless you can place enough of them.
Okay, so what is a PodGroup actually?
There are three objects, and mixing them up is how this topic gets worse than it is.
Workload is the template. Scheduling policy, not a running job.
PodGroup is the runtime unit. This is what the scheduler actually waits on.
Each Pod points at that unit with spec.schedulingGroup.podGroupName.
A PodGroup with a gang policy carries a minCount. If the cluster cannot place at least that many, none of them bind. They go to the unschedulable queue, and other work is allowed to run in the meantime.
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
name: inference-workers
spec:
schedulingPolicy:
gang:
minCount: 8
apiVersion: v1
kind: Pod
metadata:
name: worker-0
spec:
schedulingGroup:
podGroupName: inference-workers
containers:
- name: worker
image: inference:v1The scheduler holds those Pods in PreEnqueue until the PodGroup exists and at least minCount Pods have been created. Then it tries to place the group as one decision. Enough feasible placements, and they bind. Not enough, and none of them do.
Ugly name? A little. Ambiguous? Much less than “it will schedule when it can.”
If you do not want to hand-write PodGroups, the Job API grew an explicit .spec.scheduling field for this. gang: {} with no minCount defaults to parallelism.
apiVersion: batch/v1
kind: Job
metadata:
name: distributed-training-job
spec:
parallelism: 8
completions: 8
scheduling:
schedulingPolicy:
gang: {}
disruptionMode:
all: {}
template:
spec:
restartPolicy: Never
containers:
- name: trainer
image: train:v1
Why inference made me care
I did not come here from a batch-Job blog post. I came here from inference.
Prefill is the part where the model splits the prompt in parallel and fills the KV cache. Decode is the part where it emits one token at a time and has to keep reading that cache. Different shapes. Same constraint: a replica that is only half placed is not “degraded.” It is stuck.
A PodGroup with minCount is enough to say that. The hierarchical version is still alpha. So I would not start there.
minCount can change in 1.37 without tearing down Pods that are already bound. That is what makes this usable for serving, not only for a training Job with a fixed size.
Why we should care
The Norway problem of scheduling is idle GPUs that look busy.
Wrongly placed Pods are still valid Pods. They just represent a different job than the one you thought you launched. That is annoying for a CPU worker. It is miserable when the thing they are holding is an H100 (cost-wise, that is).
The other reason it matters now is that more of these workloads are generated by controllers. A human staring at kubectl get pods can see that five of eight is a lie. A controller creating hundreds of ranks cannot. The scheduler has to be the thing that refuses the partial placement.
This also does not ask you to replace kube-scheduler with Volcano tomorrow. The KEP linked earlier details how fairness and multi-queue stay with Kueue and friends. What really changes is that in-tree is the all-or-nothing rule, so those systems have a native hook instead of each inventing a Permit plugin.
Why I am willing to try it
I am not about to turn this on in production because of this blog. That would be a different kind of headache.
What I am willing to do is the cheap experiment.
Turn on GenericWorkload. Make a PodGroup that needs four Pods. Give the cluster room for three. If three of them still bind, the gang did not work. If none of them bind, it did.
That is the part that got me past “guess we are all in a gang now.” It is not a new scheduler but more of an opinionated way to place a shape of workload we are already running.
The other reason I will try it, I have been reading enough inference internals lately
I still do not want this as the default. Most Deployments should keep scheduling Pod by Pod. The KEP even keeps basic as that path.
Using gang for GPU workers, for training Jobs, and for anything that cannot do useful work at replicas - 1, feels like the right place to start.
Where this leaves you
If you are on Kubernetes 1.37, read the workload-aware scheduling blog and then the gang scheduling docs. That is the whole briefing.
If you want to poke it, enable GenericWorkload on a cluster that is allowed to be wrong, create a PodGroup, and watch whether Pending means “not yet” or “not as a group.”
If you already run Volcano or Kueue, this is not a replacement this week. And if your cluster is still going to schedule Pods one at a time until the heat death of the GPU pool, that is fine.
Kubernetes is not making this the default, and it will keep placing the Deployments you already have. The win is optional, which is exactly why I am willing to poke at it.
🌍IN THE ECOSYSTEM
Kubernetes v1.37: Advancing Workload-Aware Scheduling — the official release blog: PodGroup and Workload to beta, gang scheduling, workload-aware preemption, CompositePodGroup, and the Job .spec.scheduling field.
Gang Scheduling — how the plugin actually behaves: PreEnqueue, minCount, the unschedulable queue, and the training vs inference split on CompositePodGroup.
KEP-4671: Gang Scheduling using Workload Object — why this exists, why PodGroup is a runtime object instead of a field on Workload, and the line about implementing gang scheduling outside kube-scheduler at least four times.
⏱️UNTIL NEXT TIME
That’s it for this issue. If you try GenericWorkload this week, reply and tell me whether the gang looks like a fix or like a Permit plugin in a cheap disguise. I am especially interested if you try it on GPU workers and it either saves you from a half-placed replica or just sits in Pending forever.
Know an engineer who will find this helpful? Share this link with them
Jubril Oyetunji
CTO, EverythingDevOps