Skip to main content

Batching

Grouping several small units of work into one task so that scheduling overhead does not dominate the time spent doing useful work.

A task that takes a millisecond spends most of its life being scheduled. Batching a hundred such units into one task changes the ratio decisively.

Performance & optimizationDistributed compute

Related terms

  • Task: One unit of distributed work: a single call to a function decorated with the distribute decorator. Tasks are stateless, so each call starts fresh.
  • Partition: One independently processable chunk of a dataset. Partition size is the main lever on how well a workload parallelizes.
  • Scheduling overhead: The fixed cost of dispatching a task and returning its result. When it approaches the cost of the task body, adding parallelism stops helping.