Skip to main content

Requesting CPUs and GPUs per call

options() sets what a particular call needs:

@eugo.hpc.distribute
def train_step(batch):
    return run_step(batch)

# This call wants a GPU and four cores
future = train_step.options(num_cpus=4, num_gpus=1)(batch)

Why the call site, not the definition​

The same function is often invoked in different circumstances. A quick smoke test over ten records does not need a GPU; the production run over ten million does.

Putting requirements on the definition would force the expensive shape everywhere:

# Defined once
@eugo.hpc.distribute
def process(chunk):
    return transform(chunk)

# Cheap validation pass: default resources
sample = eugo.hpc.get([process(c) for c in chunks[:10]])

# Full run: heavier resources
futures = [process.options(num_cpus=8)(c) for c in chunks]

One definition, two resource profiles.

What the scheduler does with it​

The scheduler places each task on a node that can satisfy its request. A task asking for a GPU only runs on a node that has one free; if none does, it waits.

That waiting is the failure mode worth recognizing. Tasks that queue indefinitely usually mean a request nothing can satisfy, most often a GPU request on a cluster whose nodes have none.

Over-requesting is expensive twice​

# Bad: every task demands a GPU, including the ones doing string parsing
futures = [step.options(num_gpus=1)(x) for x in items]

Two costs:

  1. Queueing. CPU-only tasks wait behind GPU availability they never needed. Throughput drops even though the cluster has idle CPU capacity.
  2. Money. GPU nodes carry a premium. Paying it for work that never touches the GPU is pure waste.

In a mixed pipeline, request GPUs only for the stages that use them:

# Parsing: CPU
parsed = [parse.options(num_cpus=2)(f) for f in files]

# Numeric transform: GPU earns its cost here
transformed = [heavy_math.options(num_gpus=1)(p) for p in parsed]

Fractional GPUs​

Several small tasks can share one device:

futures = [infer.options(num_gpus=0.25)(x) for x in items]

Four such tasks fit on one GPU. Useful for inference, where a single request rarely saturates the device. Be careful with memory. Fractional allocation divides scheduling, not VRAM, so four tasks each wanting most of the card will fail.

Sensible defaults​

Without options(), a task gets a default allocation of roughly one CPU and no GPU. That is correct for most work, and reaching for options() on every call is a sign of over-tuning.

Set it when you know a specific reason: the task is multi-threaded internally, it needs a GPU, or it needs more memory than one core's share.

Verify rather than assume​

Requesting a GPU is not proof one was used. Confirm inside the task:

@eugo.hpc.distribute
def check():
    import torch
    return {
        "cuda": torch.cuda.is_available(),
        "device": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None,
    }

print(eugo.hpc.get(check.options(num_gpus=1)()))

GPU acceleration on Eugo covers verification and transfer costs in depth, including the cases where a GPU is measurably slower than its own machine's CPU.

Video: Requesting GPUs per task. The same material, with a transcript.