Skip to main content

Confirming your work ran on a GPU

Requesting a GPU guarantees the hardware. It does not guarantee your work used it, or that using it helped. Three separate things, each worth checking.

1. Was hardware allocated?​

@eugo.hpc.distribute
def gpu_check():
    import torch
    return {
        "available": torch.cuda.is_available(),
        "count": torch.cuda.device_count(),
        "name": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None,
    }

print(eugo.hpc.get(gpu_check.options(num_gpus=1)()))

available: False with num_gpus=1 requested means the task did not land on a GPU node. Check plan availability on the dashboard.

2. Did your work use it?​

For PyTorch, tensors carry their device:

@eugo.hpc.distribute
def where_did_it_run(array):
    import torch
    t = torch.from_numpy(array)
    result = expensive_matrix_work(t)
    return {"input": str(t.device), "output": str(result.device)}

cpu for both, on a node where CUDA was available, means the work stayed on the host. Usually the array was too small for offloading to pay, which is the runtime making a correct decision.

3. Was it actually faster?​

The check people skip, and the only one that matters commercially.

import time

@eugo.hpc.distribute
def timed(array):
    start = time.perf_counter()
    result = expensive_matrix_work(array)
    return {"seconds": time.perf_counter() - start, "shape": result.shape}

cpu = eugo.hpc.get(timed(array))
gpu = eugo.hpc.get(timed.options(num_gpus=1)(array))

print(f"CPU {cpu['seconds']:.3f}s   GPU {gpu['seconds']:.3f}s   "
      f"speedup {cpu['seconds'] / gpu['seconds']:.1f}x")

Run each a few times and take the median. First-call timings include device initialization and are misleading.

GPU calls are asynchronous

CUDA operations queue rather than block. Timing around them without synchronizing measures how fast you submitted work, not how long it took.

result = expensive_matrix_work(t)
torch.cuda.synchronize()          # wait for completion before stopping the clock
elapsed = time.perf_counter() - start

Missing synchronize() produces impossibly good numbers. If a speedup looks too good, this is usually why.

Expect a slowdown sometimes​

A speedup below 1x is a legitimate result, not a bug. From the illustrative chart data — figures pending verification, so read the ratio as a shape rather than as a measurement:

  • Matmul at 256²: about 2.7x slower on the GPU than on the same machine's CPU
  • Elementwise over 10K elements: about 17x slower, same comparison

Transfer cost exceeded compute saved. Finding this is the check working: you now know to leave that stage on CPU and stop paying the GPU premium for it.

What to record​

For each stage you consider accelerating:

Value
CPU time (median of 3)
GPU time (median of 3, synchronized)
Speedup
Verdict: keep GPU?

Without those numbers, "we use GPUs" is an architecture claim rather than a performance one.

Next​

Transfer costs and when GPUs lose goes into why the boundary sits where it does, and how to shift it.