import { TransferCost } from '@site/src/components/diagrams/TransferCost';

# Transfer costs and when GPUs lose

A GPU is fast at arithmetic and connected to the rest of the machine by a comparatively narrow pipe.
Every offload pays for crossing it, twice.

## Arithmetic intensity

The quantity that decides everything:

```
arithmetic intensity  =  operations performed / bytes moved
```

High intensity means many operations per byte, and the GPU wins easily. Low intensity, and you have
built an expensive data-copying machine.

This is why array *size* is a poor predictor on its own:

:::warning Figures pending verification
The CPU/GPU figures below are **illustrative** — no capture backs them. They are here to show
*when* offloading pays and when the transfer eats the gain, not to state what any particular
machine does. Do not quote them as Eugo performance figures; measure your own workload.
:::

| Operation | Data | Ops per element | GPU verdict |
| --- | --- | --- | --- |
| Matmul, n=8192 | 256 MB | ~8192 | ~50x faster |
| Sum, 512M elements | 2 GB | 1 | ~1.8x, transfer-dominated |

The reduction moves eight times more data and does one operation per element. The matmul moves less and
does thousands. Intensity, not volume.

## The break-even point

Below some size, transfer overhead swamps the computation:

| Matmul size | CPU | GPU | Verdict |
| --- | --- | --- | --- |
| 256² | 3.1 ms | 8.4 ms | CPU wins |
| 8192² | 4820 ms | 96 ms | GPU wins decisively |

Somewhere between sits the crossover. It depends on the operation and the hardware, which is why
[measuring](./03-verifying-gpu-use.mdx) beats reasoning about it.

<TransferCost />

## Moving the break-even point

Three ways to make transfers pay better.

### Batch small operations together

```python
# Bad: 1000 transfers for 1000 tiny operations
results = [small_op.options(num_gpus=0.1)(x) for x in items]

# Better: one transfer, one large operation
batched = np.stack(items)
results = batch_op.options(num_gpus=1)(batched)
```

One transfer of 1000× the data costs far less than 1000 transfers, because the fixed per-transfer
overhead is paid once.

### Chain operations on device

Each round trip to the host costs a transfer pair:

```python
# Bad: three round trips
a = step_one.options(num_gpus=1)(data)
b = step_two.options(num_gpus=1)(eugo.hpc.get(a))
c = step_three.options(num_gpus=1)(eugo.hpc.get(b))

# Better: one transfer in, one out
@eugo.hpc.distribute
def pipeline(data):
    return step_three(step_two(step_one(data)))   # stays on device
```

### Keep reused data resident

If many calls use the same large array, an [actor](/lesson/distributed-python-with-eugo-hpc/actors)
holds it on device across calls rather than re-uploading:

```python
@eugo.hpc.distribute
class Resident:
    def __init__(self, weights):
        import torch
        self.weights = torch.from_numpy(weights).cuda()   # uploaded once

    def apply(self, batch):
        return (batch_to_device(batch) @ self.weights).cpu().numpy()
```

## Precision

Halving the bytes halves the transfer:

```python
data32 = data.astype(np.float32)     # half of float64
```

`float32` is standard for GPU work and sufficient for most machine learning and much scientific
computing. Check that your problem tolerates it. Some numerical methods do not, and silently losing
precision is worse than a slow correct answer.

## When a GPU cannot help at all

**I/O-bound work.** If most of the time is waiting on [object storage](/glossary/object-storage), the GPU sits idle regardless. Fix
the I/O first. See [when I/O is the bottleneck](/lesson/scaling-a-real-workload/io-bound-vs-cpu-bound).

**Branchy control flow.** GPUs want uniform work across many elements. Heavy per-element branching
serializes the divergent paths.

**Small data, whatever you do.** Below break-even, no amount of tuning makes the transfer worthwhile.

## The decision, in order

1. Is the work numeric array computation? If not, stop.
2. Is arithmetic intensity high, meaning many operations per byte?
3. Is the data large enough to clear the transfer cost?
4. Have you [measured](./03-verifying-gpu-use.mdx) it, synchronizing properly?
5. Is the [speedup](/glossary/speedup) worth the node premium?

Five yeses means use a GPU. A no anywhere means leave that stage on CPU and spend the effort elsewhere.

---

Source: https://university.eugo.io/lesson/gpu-acceleration-on-eugo/transfer-costs