Requesting GPUs for a task

Automatic offloading needs a GPU to offload to. That is a scheduling request:

@eugo.hpc.distribute
def heavy_math(array):
    return expensive_matrix_work(array)

future = heavy_math.options(num_gpus=1)(array)

The request is a constraint

num_gpus=1 tells the scheduler this task requires a node with a free GPU. Until one exists, the task waits.

That produces a recognizable symptom: tasks submitted, none running. Almost always a resource request nothing on the cluster can satisfy, commonly a GPU request on a cluster provisioned without GPU nodes. Check GPU availability for your plan on the dashboard before designing around it.

Fractional GPUs

A single inference call rarely saturates a modern device. Several tasks can share one:

futures = [infer.options(num_gpus=0.25)(x) for x in items]

Four tasks per GPU, four-way parallelism, one device.

Fractional allocation divides scheduling, not memory

num_gpus=0.25 lets four tasks co-schedule on one device. It does not partition VRAM. Four tasks each wanting 20 GB on a 24 GB card will fail with out-of-memory errors, not queue politely.

Size fractions by memory footprint, not by convenience.

Mixed pipelines

Request GPUs stage by stage:

# Parse: CPU work, no GPU
parsed = [parse.options(num_cpus=2)(f) for f in files]

# Transform: high arithmetic intensity, GPU earns its cost
transformed = [heavy_math.options(num_gpus=1)(p) for p in parsed]

# Summarize: cheap, CPU
summary = [summarize.options(num_cpus=1)(t) for t in transformed]

What over-requesting costs

# Bad: every task demands a GPU, including string parsing
futures = [step.options(num_gpus=1)(x) for x in items]

Two separate costs:

Throughput. CPU-only tasks queue behind GPU availability. The cluster has idle CPU capacity and your run is slower than it would be without the request.

Money. GPU nodes carry a premium. Paying it for work that never touches the device is waste with no upside.

CPUs matter too

num_cpus is worth setting when a task is internally multi-threaded, as many NumPy operations are. Without it a task gets roughly one core's share and its threads contend:

future = numpy_heavy.options(num_cpus=8)(data)

Requesting more cores than the task can use is the same mistake as over-requesting GPUs, in a smaller denomination.

Combining both

future = train_step.options(num_cpus=4, num_gpus=1)(batch)

Typical for training: the GPU does the arithmetic while CPU cores handle data loading and augmentation feeding it.

Then verify

A request is not proof of use. Confirming your work ran on a GPU covers how to check.