Workload readiness checklist

Measure how long one partition, file, or record batch takes on one machine. Every sizing decision downstream depends on this number, and guessing it wrong is the most common reason a cluster underdelivers.

Establish how many units exist. Serial time is roughly units multiplied by per-unit time, and that product is the ceiling on what distribution can save.

Profile how much of the run is spent reading and writing versus calculating. If I/O dominates, adding workers adds waiting rather than throughput.

Each unit must produce its result without needing another unit’s output. Anything sequential needs restructuring first: a running total, a dependency chain.

Look for a global counter, a shared cache, or a file every worker writes to. Shared mutable state is what turns a parallel run into a race condition.

Know how partial results merge, and whether that combine step is itself expensive enough to become the new bottleneck.

Data on a single machine’s local disk is not visible to the cluster. Inputs need to be in object storage or workspace-persistent storage.

A columnar format lets a worker read only the fields it uses. On a wide table, that alone can cut read volume by an order of magnitude.

Too small and scheduling overhead dominates; too large and the final few tasks leave most workers idle. Aim for units that take seconds, not milliseconds.

Decide what runtime would make this worth doing. Without a target, "faster" has no stopping point and tuning never ends.

Worker count multiplied by expected wall-clock time is your compute spend. Check it against the plan limits on your dashboard before launching.