Distributed compute
Spreading work across a cluster with tasks, actors, and futures.
Courses
Distributed Python with eugo.hpc
Turn ordinary Python into distributed work with a decorator, then scale it across a cluster.
How automatic optimizations work
Eugo rewrites and tunes your code as it runs. Here is what it does, and how to help it.
Scaling a real workload
Take a single-machine geospatial pipeline to a cluster, and find out where the time actually goes.
Videos
Coming soonLaunch your first cluster
From an empty notebook to a running cluster and a completed task, including the blocking mistake almost everyone makes on their first fan-out.
Coming soonThe distribute decorator, explained
What @eugo.hpc.distribute actually changes about a function, and why the call site stays clean.
Coming soonWorking with futures without blocking
The most common performance mistake in distributed code, and the two-line habit that avoids it.
Coming soonActors for stateful work
When a task is the wrong shape: keeping an expensive object alive across many calls.
Coming soonParallel pandas in practice
Split a dataframe workload across workers without rewriting your analysis.
Coming soonSeven terabytes in three minutes
A real geospatial pipeline: 192,000 satellite files, resampled and written, start to finish.
Checklists and guides
Workload readiness checklist
Verify a workload is worth distributing before you spend cluster time on it.
Performance tuning checklist
Work through the usual causes when a distributed run is slower than expected.
Quick guide: data scientist
The short version of what a data scientist needs on day one: where code runs, where data has to live, and the one habit that keeps a cluster busy.
Vocabulary
The terms this subject uses, defined.
- High-performance computing (HPC) : Running work across many machines at once so it finishes far sooner. The technique behind weather models, genomics, and large-scale simulation.
- Distributed computing : Splitting one job across several machines that work on it simultaneously. The core technique of HPC, and what the distribute decorator gives you in Python.
- Job scheduler : The queueing system on a traditional HPC cluster that decides when a submitted job runs. Eugo has no queue: you launch a cluster and dispatch work to it.
- Parallel computing : Many calculations at once rather than one after another. Two scales on Eugo: across machines in a cluster, and inside a CPU core through vectorization.
- Actor : A distributed object that holds state across many calls. Created by applying the distribute decorator to a class rather than a function.
- Batching : Grouping several small units of work into one task so that scheduling overhead does not dominate the time spent doing useful work.
- Cluster : A set of machines, one head node and one or more compute nodes, launched together to run distributed work.
- Compute node : A machine in a cluster that executes tasks. Compute nodes do the work; the head node coordinates it.
- distribute decorator : The eugo.hpc decorator that turns a function into a distributed task, or a class into an actor. A decorated call returns a future instead of a value.
- EugoHPC Manager : The panel inside EugoIDE used to launch, monitor, and shut down clusters.
- Fan-out : Launching many independent tasks at once, then waiting on all of their results together.
- Future : A handle to a result that does not exist yet. Calling a distributed function returns a future immediately, while the work happens elsewhere.
- Blocking : Waiting for a result before continuing. In distributed code, blocking too early is the most common cause of a run that gains no speedup.
- Head node : The machine in a cluster that coordinates work: it schedules tasks onto compute nodes and tracks their results.
- Long tail : The situation where most tasks finish quickly but one or two run much longer, so the whole run waits on them while workers sit idle.
- Partition : One independently processable chunk of a dataset. Partition size is the main lever on how well a workload parallelizes.
- Scheduler : The component on the head node that decides which compute node runs each task, honoring the resources that task requested.
- Scheduling overhead : The fixed cost of dispatching a task and returning its result. When it approaches the cost of the task body, adding parallelism stops helping.
- Serialization : Converting Python objects into bytes so they can be sent to another machine. Large task arguments are serialized and copied to every worker that needs them.
- Speedup : How many times faster a distributed run is than the same work on one worker. Ideal speedup equals worker count; real speedup falls short of it.
- Straggler : A single task that runs far longer than its peers and holds up completion of the whole run.
- Task : One unit of distributed work: a single call to a function decorated with the distribute decorator. Tasks are stateless, so each call starts fresh.
- Worker : A process on a compute node that executes tasks. Worker count, not node count, is what determines how many tasks run at once.
- Shared state : Data that more than one task reads or writes. Shared mutable state is what turns a parallel run into a race condition.
- Independent work : Units of work that can each produce their result without needing another unit’s output. A precondition for distributing anything.