Skip to main content

Worker

A process on a compute node that executes tasks. Worker count, not node count, is what determines how many tasks run at once.

Worker count rather than node count is what determines how many tasks run at once, which is why a cluster can be fully allocated and still underused if the work is not partitioned finely enough.

Distributed compute

Related terms

  • Compute node: A machine in a cluster that executes tasks. Compute nodes do the work; the head node coordinates it.
  • Task: One unit of distributed work: a single call to a function decorated with the distribute decorator. Tasks are stateless, so each call starts fresh.
  • Scheduler: The component on the head node that decides which compute node runs each task, honoring the resources that task requested.
  • Distributed computing: Splitting one job across several machines that work on it simultaneously. The core technique of HPC, and what the distribute decorator gives you in Python.