Launch, watch, and shut down a cluster
The panel on the right of EugoIDE controls clusters. This lesson runs one complete lifecycle: launch, dispatch, watch, stop. Run it once and the sequence is muscle memory before you need it on real work.
Use two workers. Two teaches the same mechanics as sixty-four at a fraction of the cost.
1. Launch it
Open the panel, choose two workers, give it a name, and start it.
Watch the states as it comes up. The head node appears first, then compute nodes register with it. You are waiting for ready.
Do not dispatch work before then. Tasks submitted to a partially-formed cluster either queue or fail, and the failure does not say "the cluster was not ready yet."
"The 32-worker one" stops being a useful identifier the moment a second cluster exists.
2. Prove the work lands somewhere else
In a notebook:
import eugo.hpc, os, socket
@eugo.hpc.distribute
def where_am_i(i):
return f"task {i} on {socket.gethostname()} (pid {os.getpid()})"
for line in eugo.hpc.get([where_am_i(i) for i in range(8)]):
print(line)
Read the hostnames. They are not your session's hostname. Compare against the hostname you noted
in the interface tour.
You should see more than one distinct host across those eight tasks. Eight lines is enough to show the cluster doing the thing you launched it for.
3. Watch the panel while something runs
Dispatch something slower so there is time to look:
import time
@eugo.hpc.distribute
def slow(i):
time.sleep(3)
return i
futures = [slow(i) for i in range(24)] # returns immediately
results = eugo.hpc.get(futures) # now watch the panel
While that runs, the signal worth watching is distribution across workers.
Work spread across both workers is healthy. Work concentrated on one is the symptom you need to recognize. The next section produces it on purpose.
4. Cause the failure on purpose
Ask for a resource the cluster does not have:
futures = [slow.options(num_gpus=1)(i) for i in range(4)]
results = eugo.hpc.get(futures) # this will sit there
On a CPU-only cluster these tasks queue forever. No error, no warning, just a cell that never finishes.
That is what "tasks queue but never run" means: a resource request no node can satisfy. Recognizing it saves an afternoon, because every instinct says to look at your code first.
Interrupt the cell.
5. Shut it down
Stop the cluster from the same panel.
Nothing shuts a cluster down for you. This is the largest avoidable expense in most organizations.
Also worth knowing: a cluster outlives your notebook. Restarting your kernel does not stop it. You will need to reconnect, and it bills throughout.
The habit that avoids all of this: launch, run, shut down, in one sitting. The clusters that leak money are the ones started on a Friday afternoon.
Sizing, for when it is real work
Size follows the work, not the ceiling. Start from how many independent units exist and how long one takes:
- 1,000 units at 2 seconds each ≈ 33 minutes serially
- 16 workers ≈ 2 minutes, minus overhead
Past the point where per-task overhead approaches per-task work, more workers stop helping. The cluster sizer does this arithmetic and shows where the curve flattens for your shape of workload.
Diagnosing the three things that go wrong
| Symptom | Usually means |
|---|---|
| Cluster never reaches ready | at your plan's concurrency limit (check the dashboard) |
| Tasks queue but never run | a resource request no node satisfies (you just did this) |
| One worker doing everything | same cause, usually a GPU request |
What you now know
- The full lifecycle, run once end to end.
- That tasks genuinely execute on other machines, verified by hostname rather than asserted.
- What a resource request nothing can satisfy looks like, because you caused it deliberately.
- That a cluster outlives your kernel and bills until stopped.
Next: watching a run and reading the logs. It covers telling a healthy distributed run from a stuck one, and finding errors that happened inside a task.
Video: Sizing a cluster for your workload (same material, with a transcript).