Consider two places a scheduler could put the same inference job. Both offer the same model of GPU. Both sit in the same network pod, with the same distance to the job’s data and peers. On every input a typical scheduler scores, they are interchangeable.

They are not interchangeable. One rack sits on a power branch with room to spare. The other sits on a branch already running near its limit, in a cooling zone that is working hard. Put the job in the second spot and, when demand peaks, a power or thermal limit is more likely to intervene — clocks come down, and the job delivers less useful work from the same hardware. Put it in the first and it runs as specified.

That example is illustrative, not measured. But it isolates the question this post is about. When two placements both satisfy a workload’s resource and locality requirements, does anything in the decision account for how power and thermal conditions will affect what the workload actually produces?

What schedulers and power controllers already do

Most GPU schedulers in production solve hard, essential problems well. Kubernetes’ default scheduler filters nodes for feasibility, then scores them on requested resources — what pods declare, rather than what they measurably use — together with affinity, taints, image locality, and topology spread. Its resource-fit scorer favors nodes with more free capacity by default, and operators can switch it to bin-packing or a custom ratio. Slurm orders work by priority and backfills around it. Volcano adds gang scheduling, dominant resource fairness, bin-packing, and NUMA- and network-topology-aware placement. NVIDIA’s Run:ai scheduler and the open-source KAI scheduler built from it focus on GPU utilization, fractional GPUs, hierarchical fair-share quotas, and NVLink-aware placement. Kueue primarily manages admission and quota, with topology-aware features that also shape where pods land.

Keeping a distributed job together, sharing a fleet fairly, and holding collective traffic inside a fabric domain all matter, and several of those choices improve energy efficiency even though energy is never scored. A well-packed, well-localized job usually wastes less.

Power is present too, mostly through measurement and limits. Slurm’s official documentation describes suspending idle nodes and collecting energy data per node — and states plainly that on shared nodes, the per-job energy it reports “will not reflect the real energy consumed by the jobs.” NVIDIA describes coordinating power, cooling, and workload behavior across racks and clusters, and running systems at Max-Q, “the point of highest performance per watt.”

And power-aware placement has a research history. The Linux kernel’s energy-aware scheduling uses a platform-supplied energy model to choose CPU placements predicted to reduce energy while preserving performance — on asymmetric processors such as Arm big.LITTLE, and only while CPUs are not over-utilized. Microsoft’s TAPAS uses historical temperature and power data to place GPU inference VMs within cooling and power constraints. PowerFlow models throughput and energy for training jobs and adjusts allocation, configuration, and placement under an energy budget.

So the idea is not new, and it has been demonstrated. Three separate questions sit behind that: whether an approach has been shown to work, whether mainstream tools offer it in a usable form, and whether an operator can deploy it reliably across its own workloads and hardware. The gap is mostly in the second and third.

The coordination problem that remains

The clearest statement of the gap comes from an operator. In describing the provisioning and operation of a 100 MW-scale AI cluster, Meta’s engineers write: “Our scheduler optimizes placement based on network topology alone.” The power delivery hierarchy, they note, “is a separate, overlapping tree.” Their research wishlist asks for a scheduler that maximizes aggregate throughput subject to power capacity limits at each level of that tree.

The same paper shows the other direction already working. Its power-capping system, Dimmer, “integrates scheduler metadata, such as the total size and resource distribution of jobs,” so that capping decisions respect job structure. Power control has become scheduler-aware. Placement, in their system and in most others, has not become power-aware.

“Placement decides which hardware, paths, neighbors, and power domains a workload meets. Power limits decide how it runs there. Either one chosen alone can miss the better pair.”

The two levers depend on each other. A power limit changes a workload’s operating point, and that can improve energy efficiency on its own — a memory-bound job often loses little throughput at a lower limit. Placement changes the assignment itself: which silicon, which neighbors, which electrical and thermal headroom. The best limit for a job depends on where it runs, and the best place for a job depends on what limit it will run under. Evaluating either alone can leave a better combination unexamined.

Which objective, measured where

“Performance per watt” sounds precise, but the right objective depends on the operating problem. A facility with a binding power limit and enough demand wants maximum useful throughput within that limit. A fixed job with a deadline wants minimum total energy while meeting it. An inference service with latency targets wants minimum energy or cost at the required service quality. An operator serving different paying tenants wants the most economic value while honoring contracts.

These lead to different decisions. Using deliberately illustrative numbers: configuration A delivers 100 requests per second at 400 watts, and configuration B delivers 90 at 250 watts. B is more energy efficient. A delivers more from one installed device. Which is better depends on demand, latency requirements, available hardware, and whether the power B frees can be used for something else. Lower instantaneous watts do not by themselves mean lower energy, either — a tighter limit that stretches runtime enough can raise total energy.

For the power-constrained operator, a defensible formulation is: maximize useful output while meeting workload service requirements, preserving the chosen fairness policy, and staying within electrical and thermal limits. That still needs three definitions. Useful output: inference results at a specified quality and latency, or training progress toward a target. The energy boundary: GPU, server, rack, or facility. The decision horizon: initial placement, periodic reallocation, or request routing. Without them, “performance per watt” leaves the actual decision ambiguous.

Why this is hard to do well

The obstacles are practical, and they explain why research results have not become a standard feature.

Models have to cover the fleet. Workload energy models exist. The difficulty is coverage and durability: they must span the relevant combinations of workload, hardware, and configuration, and stay accurate as software, models, and conditions change.

Attribution is shared. Per-GPU power is readily measured. The harder part is assigning host, network, and cooling energy to individual workloads when they share infrastructure — the same issue Slurm’s documentation flags for shared nodes. Perfect attribution is not a prerequisite for every useful decision; aggregate measurement and sufficiently reliable prediction can be enough.

History does not reveal causes by itself. Low throughput might reflect a changed workload, resource pressure, or an environmental limit, and those call for different responses. Separating them takes evidence, not just a record of what happened.

Signals move at different speeds. Device power can be sampled every second or faster; facility measurements arrive at their own cadences, with their own delays and aggregation, and thermal systems respond on their own physical timescales. Aligning them is a problem in itself.

Utilization alone does not establish efficiency. NVIDIA’s GPU utilization reports the share of time any kernel was executing; SM occupancy measures how many warps are resident relative to capacity. Both are useful inputs, and neither by itself says how much useful output was produced per joule.

The fleet is not the sum of its jobs. Giving one job its preferred placement can deny another a larger benefit, so per-job optimization does not automatically optimize the facility.

Freedom to move is limited. Hardware compatibility, data locality, reservations, and the cost of migration can remove options that look attractive on paper.

Where the value sits

The commercial case needs the same care. Operators selling GPU-hours are not directly paid for useful output. Better coordination creates value when it shows up as capacity that can actually be sold, better service quality, stronger retention, or wider margins — and the first of those depends on demand being there to absorb what is freed.

Where power is the binding constraint, those conditions are common. A facility that gets more useful output from the same electrical envelope has, in effect, more to sell.

This is the space Serra Labs works in: workload-aware optimization of AI infrastructure, where how a workload behaves informs how it is sized and placed. We think the most useful progress will come less from new algorithms than from making workload prediction, network topology, and facility constraints usable together, in a form operators can trust.

A question to put to your stack

Most stacks report power in some form, and the better ones report it well. The more useful test is this:

When two placements meet a workload’s resource and locality requirements, can your system estimate how power and thermal conditions will affect useful output — and use that estimate in its decision?