# Performance tuning checklist

Work through the usual causes when a distributed run is slower than expected.

Source: https://university.eugo.io/resources/performance-tuning-checklist

---

- Kind: Checklist
- Topics: performance, distributed-compute, gpu-acceleration

## Check the obvious mistakes first

- **Make sure you are not blocking inside a loop** — Calling get() on each future as you create it serializes the whole run. Collect all futures first, then wait on the list once. This single mistake accounts for more missing speedup than everything else combined.
- **Verify tasks are reaching more than one worker** — Check the run summary for task distribution. Work concentrated on one node usually means a resource request no other node can satisfy.
- **Look for an accidental sequential dependency** — If task N takes task N-1’s result as an argument, the cluster runs them in order no matter how many workers are available.

## Examine the task shape

- **Measure per-task overhead against per-task work** — Scheduling a task is not free. When the body takes milliseconds, overhead dominates. Batch several units into one task instead.
- **Watch for a long tail** — If one partition is much larger than the rest, the run finishes when that one does. Split uneven partitions rather than adding workers.
- **Check what you are sending to each task** — Large arguments get serialized and copied to every worker. Pass a reference to shared data rather than the data itself.

## Look at resources

- **Confirm GPU work actually ran on a GPU** — Automatic offloading is conditional. Small arrays stay on the CPU because the transfer would cost more than it saves. Verify rather than assume.
- **Right-size resource requests** — Requesting a GPU for every task in a mixed pipeline makes CPU-only tasks queue behind GPU availability they never needed.
- **Reconsider whether the cluster is the constraint** — If workers are idle and throughput is flat, the limit is elsewhere: object storage, a single input file, or a downstream write.