Skip to main content

Long tail

The situation where most tasks finish quickly but one or two run much longer, so the whole run waits on them while workers sit idle.

Usually caused by uneven partition sizes. The fix is splitting the oversized partitions, not adding workers.

Performance & optimizationDistributed compute

Related terms

  • Partition: One independently processable chunk of a dataset. Partition size is the main lever on how well a workload parallelizes.
  • Speedup: How many times faster a distributed run is than the same work on one worker. Ideal speedup equals worker count; real speedup falls short of it.
  • Straggler: A single task that runs far longer than its peers and holds up completion of the whole run.