# What gets offloaded, and what does not

Eugo inspects the operations in your code and moves computationally intensive array work onto a GPU
when one is available **and the transfer is worth paying for**.

That second clause is what this lesson is about.

## The arithmetic

Using a GPU means moving data to device memory and results back. That transfer has a fixed cost,
independent of how clever the computation is.

Offloading pays only when:

```
compute_saved  >  transfer_cost
```

For a large matrix multiply, compute saved is enormous and transfer is a rounding error. For a
thousand-element array, transfer is the entire cost and there is nothing to save.

## Illustrative numbers

:::warning Figures pending verification
The CPU/GPU figures below are **illustrative** — no capture backs them. They are here to show
*when* offloading pays and when the transfer eats the gain, not to state what any particular
machine does. Do not quote them as Eugo performance figures; measure your own workload.
:::

From the [GPU benchmark](/interactive):

| Operation | CPU | GPU | Transfer | Result |
| --- | --- | --- | --- | --- |
| Matmul 8192² | 4820 ms | 96 ms | 41 ms | **50x faster** |
| Elementwise, 512M | 1960 ms | 148 ms | 112 ms | **13x faster** |
| FFT 4096² | 1240 ms | 87 ms | 22 ms | **14x faster** |
| Sum, 512M | 214 ms | 119 ms | 108 ms | 1.8x, mostly transfer |
| Matmul 256² | 3.1 ms | 8.4 ms | 7.6 ms | **2.7x slower** |
| Elementwise, 10K | 0.4 ms | 6.9 ms | 6.7 ms | **17x slower** |

The bottom two rows matter most. On small data a GPU is *slower than the CPU sitting beside it in
the same machine*, and that is not a defect. The [transfer cost](/glossary/transfer-cost) is larger
than the work.

Look at the reduction row too: 512M elements is a lot of data, but summing is cheap per element, so
transfer dominates and the [speedup](/glossary/speedup) is modest. Data size alone does not decide it. **Arithmetic
intensity** does, meaning how much computation happens per byte moved.

## What offloads well

- **Matrix multiplication** on large matrices. High arithmetic intensity, the ideal case
- **Elementwise transforms** over large arrays
- **Convolutions** and FFTs
- **Neural network forward and backward passes**

## What does not

- **Small arrays**, whatever the operation
- **Cheap reductions** over large data: high bytes, low arithmetic
- **Branchy, control-flow-heavy code.** GPUs want uniform work across many elements
- **String and object manipulation.** Not numeric array work at all
- **Anything I/O-bound.** A GPU cannot accelerate waiting on [object storage](/glossary/object-storage).

## You still have to ask for the hardware

Offloading is automatic; *hardware allocation is not*. A [task](/glossary/task) without a GPU in its resource request
runs on a node that may not have one:

```python
# CPU only: nothing to offload to
future = heavy_math(data)

# Now there is a GPU available for the runtime to use
future = heavy_math.options(num_gpus=1)(data)
```

See [requesting GPUs for a task](./02-requesting-gpus.mdx).

## The practical consequence

Do not sprinkle GPU requests across a pipeline hoping for speed. Identify the stages with genuinely
high arithmetic intensity over large arrays, request GPUs for those, and leave everything else on CPU.

Requesting a GPU everywhere costs money on the nodes and throughput on the queue. CPU tasks end up
waiting behind GPU availability they never needed.

Then [verify](./03-verifying-gpu-use.mdx) rather than assume.

---

**Video:** [Automatic GPU offloading](/videos/automatic-gpu-offloading). The same material, with a transcript.

---

Source: https://university.eugo.io/lesson/gpu-acceleration-on-eugo/what-gets-offloaded