Transfer costs and when GPUs lose

A GPU is fast at arithmetic and connected to the rest of the machine by a comparatively narrow pipe. Every offload pays for crossing it, twice.

Arithmetic intensity

The quantity that decides everything:

arithmetic intensity  =  operations performed / bytes moved

High intensity means many operations per byte, and the GPU wins easily. Low intensity, and you have built an expensive data-copying machine.

This is why array size is a poor predictor on its own:

Figures pending verification

The CPU/GPU figures below are illustrative — no capture backs them. They are here to show when offloading pays and when the transfer eats the gain, not to state what any particular machine does. Do not quote them as Eugo performance figures; measure your own workload.

Operation	Data	Ops per element	GPU verdict
Matmul, n=8192	256 MB	~8192	~50x faster
Sum, 512M elements	2 GB	1	~1.8x, transfer-dominated

The reduction moves eight times more data and does one operation per element. The matmul moves less and does thousands. Intensity, not volume.

The break-even point

Below some size, transfer overhead swamps the computation:

Matmul size	CPU	GPU	Verdict
256²	3.1 ms	8.4 ms	CPU wins
8192²	4820 ms	96 ms	GPU wins decisively

Somewhere between sits the crossover. It depends on the operation and the hardware, which is why measuring beats reasoning about it.

Illustrative figures, not measurements — no capture backs them. Both rows depict one machine’s own CPU against its own GPU. The hatched portion is host-to-device transfer and back, paid on every offload regardless of how much arithmetic follows. It is drawn as a share of each GPU bar rather than measured against the axis, since the axis is logarithmic. On the large matmul it is a rounding error. On the small one it is nearly the whole bar, so there is no computation left to save. Size alone does not decide it. Arithmetic intensity, operations per byte moved, does.

Moving the break-even point

Three ways to make transfers pay better.

Batch small operations together

# Bad: 1000 transfers for 1000 tiny operations
results = [small_op.options(num_gpus=0.1)(x) for x in items]

# Better: one transfer, one large operation
batched = np.stack(items)
results = batch_op.options(num_gpus=1)(batched)

One transfer of 1000× the data costs far less than 1000 transfers, because the fixed per-transfer overhead is paid once.

Chain operations on device

Each round trip to the host costs a transfer pair:

# Bad: three round trips
a = step_one.options(num_gpus=1)(data)
b = step_two.options(num_gpus=1)(eugo.hpc.get(a))
c = step_three.options(num_gpus=1)(eugo.hpc.get(b))

# Better: one transfer in, one out
@eugo.hpc.distribute
def pipeline(data):
    return step_three(step_two(step_one(data)))   # stays on device

Keep reused data resident

If many calls use the same large array, an actor holds it on device across calls rather than re-uploading:

@eugo.hpc.distribute
class Resident:
    def __init__(self, weights):
        import torch
        self.weights = torch.from_numpy(weights).cuda()   # uploaded once

    def apply(self, batch):
        return (batch_to_device(batch) @ self.weights).cpu().numpy()

Precision

Halving the bytes halves the transfer:

data32 = data.astype(np.float32)     # half of float64

float32 is standard for GPU work and sufficient for most machine learning and much scientific computing. Check that your problem tolerates it. Some numerical methods do not, and silently losing precision is worse than a slow correct answer.

When a GPU cannot help at all

I/O-bound work. If most of the time is waiting on object storage, the GPU sits idle regardless. Fix the I/O first. See when I/O is the bottleneck.

Branchy control flow. GPUs want uniform work across many elements. Heavy per-element branching serializes the divergent paths.

Small data, whatever you do. Below break-even, no amount of tuning makes the transfer worthwhile.

The decision, in order

1. Is the work numeric array computation? If not, stop.
2. Is arithmetic intensity high, meaning many operations per byte?
3. Is the data large enough to clear the transfer cost?
4. Have you measured it, synchronizing properly?
5. Is the speedup worth the node premium?

Five yeses means use a GPU. A no anywhere means leave that stage on CPU and spend the effort elsewhere.