Skip to main content

Distributed computing

Splitting one job across several machines that work on it simultaneously. The core technique of HPC, and what the distribute decorator gives you in Python.

It only helps when the work divides into pieces that do not need each other's results. See independent work. Where it does divide, throughput scales with the number of workers until either the partitions become too small to be worth scheduling or the job turns out to be waiting on I/O rather than computing.

Distributed computeFundamentals

Related terms

  • High-performance computing (HPC): Running work across many machines at once so it finishes far sooner. The technique behind weather models, genomics, and large-scale simulation.
  • Independent work: Units of work that can each produce their result without needing another unit’s output. A precondition for distributing anything.
  • Task: One unit of distributed work: a single call to a function decorated with the distribute decorator. Tasks are stateless, so each call starts fresh.
  • Worker: A process on a compute node that executes tasks. Worker count, not node count, is what determines how many tasks run at once.
  • Parallel computing: Many calculations at once rather than one after another. Two scales on Eugo: across machines in a cluster, and inside a CPU core through vectorization.