Interactive sessions vs. clusters
Getting this distinction wrong produces code that looks distributed and runs serially, which is why it gets a lesson to itself.
Two different machines
| Interactive session | Cluster | |
|---|---|---|
| How many machines | One | Many (head + compute nodes) |
| What runs there | Your notebook | Your tasks |
| Lifetime | You open and close it | You launch and shut it down |
| Needs the other? | No | Yes, something must drive it |
How work travels
When you call a distributed function from a notebook cell:
- The session serializes the function's arguments.
- They go to the head node, which schedules the task.
- A worker on a compute node runs the function body.
- The result returns to the head node, and the session receives it when you call
get().
Every step has a cost, which is why very small tasks are inefficient. The round trip can exceed the work.
What this means in practice
Code in the notebook body runs on one machine. A plain loop is serial no matter how large your cluster is.
# Runs entirely in the session. One machine. No parallelism.
results = [expensive(x) for x in items]
Only decorated calls reach the cluster.
@eugo.hpc.distribute
def expensive_remote(x):
return expensive(x)
futures = [expensive_remote(x) for x in items] # dispatched
results = eugo.hpc.get(futures) # collected
Arguments are copied, not shared. A worker gets its own copy of what you pass. Mutating it does not affect the session's object, and passing something large to many tasks copies it many times.
The cost of large arguments
big_table = load_10gb_dataframe()
# Copies 10 GB to every worker, a thousand times over
futures = [process(big_table, i) for i in range(1000)]
If every task needs the same large object, put it in object storage and have each task read the slice it needs, or hold it in an actor. Passing it as an argument to a thousand tasks moves far more data than the computation saves.
Sizing the cluster
Worker count should follow the work, not the maximum available. Start from:
- how many independent units of work exist, and
- how long one takes.
A thousand units at two seconds each is roughly 33 minutes serial. Sixteen workers should bring that near two minutes, minus overhead.
The cluster sizer does this arithmetic interactively, including where scaling stops paying.
Both cost money while they exist
Shut the cluster down when the run finishes. Close the session when you stop for the day.