What gets offloaded, and what does not
Eugo inspects the operations in your code and moves computationally intensive array work onto a GPU when one is available and the transfer is worth paying for.
That second clause is what this lesson is about.
The arithmetic
Using a GPU means moving data to device memory and results back. That transfer has a fixed cost, independent of how clever the computation is.
Offloading pays only when:
compute_saved > transfer_cost
For a large matrix multiply, compute saved is enormous and transfer is a rounding error. For a thousand-element array, transfer is the entire cost and there is nothing to save.
Illustrative numbers
The CPU/GPU figures below are illustrative — no capture backs them. They are here to show when offloading pays and when the transfer eats the gain, not to state what any particular machine does. Do not quote them as Eugo performance figures; measure your own workload.
From the GPU benchmark:
| Operation | CPU | GPU | Transfer | Result |
|---|---|---|---|---|
| Matmul 8192² | 4820 ms | 96 ms | 41 ms | 50x faster |
| Elementwise, 512M | 1960 ms | 148 ms | 112 ms | 13x faster |
| FFT 4096² | 1240 ms | 87 ms | 22 ms | 14x faster |
| Sum, 512M | 214 ms | 119 ms | 108 ms | 1.8x, mostly transfer |
| Matmul 256² | 3.1 ms | 8.4 ms | 7.6 ms | 2.7x slower |
| Elementwise, 10K | 0.4 ms | 6.9 ms | 6.7 ms | 17x slower |
The bottom two rows matter most. On small data a GPU is slower than the CPU sitting beside it in the same machine, and that is not a defect. The transfer cost is larger than the work.
Look at the reduction row too: 512M elements is a lot of data, but summing is cheap per element, so transfer dominates and the speedup is modest. Data size alone does not decide it. Arithmetic intensity does, meaning how much computation happens per byte moved.
What offloads well
- Matrix multiplication on large matrices. High arithmetic intensity, the ideal case
- Elementwise transforms over large arrays
- Convolutions and FFTs
- Neural network forward and backward passes
What does not
- Small arrays, whatever the operation
- Cheap reductions over large data: high bytes, low arithmetic
- Branchy, control-flow-heavy code. GPUs want uniform work across many elements
- String and object manipulation. Not numeric array work at all
- Anything I/O-bound. A GPU cannot accelerate waiting on object storage.
You still have to ask for the hardware
Offloading is automatic; hardware allocation is not. A task without a GPU in its resource request runs on a node that may not have one:
# CPU only: nothing to offload to
future = heavy_math(data)
# Now there is a GPU available for the runtime to use
future = heavy_math.options(num_gpus=1)(data)
See requesting GPUs for a task.
The practical consequence
Do not sprinkle GPU requests across a pipeline hoping for speed. Identify the stages with genuinely high arithmetic intensity over large arrays, request GPUs for those, and leave everything else on CPU.
Requesting a GPU everywhere costs money on the nodes and throughput on the queue. CPU tasks end up waiting behind GPU availability they never needed.
Then verify rather than assume.
Video: Automatic GPU offloading. The same material, with a transcript.