Skip to main content

Partition

One independently processable chunk of a dataset. Partition size is the main lever on how well a workload parallelizes.

Too small and scheduling overhead dominates; too large and the final tasks leave most workers idle. Aim for units measured in seconds.

Data & I/ODistributed computePerformance & optimization

Related terms

  • Batching: Grouping several small units of work into one task so that scheduling overhead does not dominate the time spent doing useful work.
  • Long tail: The situation where most tasks finish quickly but one or two run much longer, so the whole run waits on them while workers sit idle.
  • Scheduling overhead: The fixed cost of dispatching a task and returning its result. When it approaches the cost of the task body, adding parallelism stops helping.
  • Columnar format: A file format that stores values grouped by column rather than by row, so a reader can fetch only the fields it needs.
  • Straggler: A single task that runs far longer than its peers and holds up completion of the whole run.
  • Independent work: Units of work that can each produce their result without needing another unit’s output. A precondition for distributing anything.