Skip to main content

What gets offloaded, and what does not

Eugo inspects the operations in your code and moves computationally intensive array work onto a GPU when one is available and the transfer is worth paying for.

That second clause is what this lesson is about.

The arithmetic

Using a GPU means moving data to device memory and results back. That transfer has a fixed cost, independent of how clever the computation is.

Offloading pays only when:

compute_saved > transfer_cost

For a large matrix multiply, compute saved is enormous and transfer is a rounding error. For a thousand-element array, transfer is the entire cost and there is nothing to save.

Illustrative numbers

Figures pending verification

The CPU/GPU figures below are illustrative — no capture backs them. They are here to show when offloading pays and when the transfer eats the gain, not to state what any particular machine does. Do not quote them as Eugo performance figures; measure your own workload.

From the GPU benchmark:

OperationCPUGPUTransferResult
Matmul 8192²4820 ms96 ms41 ms50x faster
Elementwise, 512M1960 ms148 ms112 ms13x faster
FFT 4096²1240 ms87 ms22 ms14x faster
Sum, 512M214 ms119 ms108 ms1.8x, mostly transfer
Matmul 256²3.1 ms8.4 ms7.6 ms2.7x slower
Elementwise, 10K0.4 ms6.9 ms6.7 ms17x slower

The bottom two rows matter most. On small data a GPU is slower than the CPU sitting beside it in the same machine, and that is not a defect. The transfer cost is larger than the work.

Look at the reduction row too: 512M elements is a lot of data, but summing is cheap per element, so transfer dominates and the speedup is modest. Data size alone does not decide it. Arithmetic intensity does, meaning how much computation happens per byte moved.

What offloads well

  • Matrix multiplication on large matrices. High arithmetic intensity, the ideal case
  • Elementwise transforms over large arrays
  • Convolutions and FFTs
  • Neural network forward and backward passes

What does not

  • Small arrays, whatever the operation
  • Cheap reductions over large data: high bytes, low arithmetic
  • Branchy, control-flow-heavy code. GPUs want uniform work across many elements
  • String and object manipulation. Not numeric array work at all
  • Anything I/O-bound. A GPU cannot accelerate waiting on object storage.

You still have to ask for the hardware

Offloading is automatic; hardware allocation is not. A task without a GPU in its resource request runs on a node that may not have one:

# CPU only: nothing to offload to
future = heavy_math(data)

# Now there is a GPU available for the runtime to use
future = heavy_math.options(num_gpus=1)(data)

See requesting GPUs for a task.

The practical consequence

Do not sprinkle GPU requests across a pipeline hoping for speed. Identify the stages with genuinely high arithmetic intensity over large arrays, request GPUs for those, and leave everything else on CPU.

Requesting a GPU everywhere costs money on the nodes and throughput on the queue. CPU tasks end up waiting behind GPU availability they never needed.

Then verify rather than assume.


Video: Automatic GPU offloading. The same material, with a transcript.