Performance tuning checklist

Calling get() on each future as you create it serializes the whole run. Collect all futures first, then wait on the list once. This single mistake accounts for more missing speedup than everything else combined.

Check the run summary for task distribution. Work concentrated on one node usually means a resource request no other node can satisfy.

If task N takes task N-1’s result as an argument, the cluster runs them in order no matter how many workers are available.

Scheduling a task is not free. When the body takes milliseconds, overhead dominates. Batch several units into one task instead.

If one partition is much larger than the rest, the run finishes when that one does. Split uneven partitions rather than adding workers.

Large arguments get serialized and copied to every worker. Pass a reference to shared data rather than the data itself.

Automatic offloading is conditional. Small arrays stay on the CPU because the transfer would cost more than it saves. Verify rather than assume.

Requesting a GPU for every task in a mixed pipeline makes CPU-only tasks queue behind GPU availability they never needed.

If workers are idle and throughput is flat, the limit is elsewhere: object storage, a single input file, or a downstream write.