Skip to main content

Launch a cluster and run something

A cluster is a head node plus one or more compute nodes, launched on demand. The head node coordinates; the compute nodes do the work.

Launch it​

Open the EugoHPC Manager panel, choose a worker count, and start the cluster. Start small. Two or four workers is plenty to learn the mechanics, and a large cluster learning the same lesson costs more.

The head node comes up first, then compute nodes register with it. Wait for the panel to report the cluster ready before dispatching work.

Your first task​

import eugo.hpc

@eugo.hpc.distribute
def square(n):
    return n * n

future = square(12)
print(future)                    # a handle, not a value
print(eugo.hpc.get(future))      # 144

Three things happened:

  1. The decorator turned square into a distributed task.
  2. Calling it scheduled work somewhere on the cluster and returned a future immediately, a handle to a result that does not exist yet.
  3. get() blocked until the result was ready and returned the value.

Printing the future rather than the result is worth doing once. It makes concrete that the call did not compute anything; it only promised to.

Now do it in parallel​

futures = [square(n) for n in range(1000)]
results = eugo.hpc.get(futures)
print(len(results), results[:5])

Every one of those thousand calls is in flight before you wait on any of them. Almost all Eugo code takes this shape: build the full list of futures, then resolve it once.

The mistake everyone makes once​

# WRONG. This runs one task at a time
results = []
for n in range(1000):
    results.append(eugo.hpc.get(square(n)))

Calling get() inside the loop blocks on each task before starting the next. The cluster sits idle while you wait, and you have paid for parallel hardware to run serial work.

The fix is moving one line out of the loop. It is frequently the difference between no speedup and near-linear speedup, and it is the first thing to check whenever a run is slower than expected.

Rule of thumb

Build all your futures first. Call get() once, on the list.

Shut it down​

When the work is finished, stop the cluster from the same panel.

An idle cluster bills exactly like a busy one. Across an organization, forgotten clusters are reliably the largest avoidable cost. See the cost optimization checklist.

Video: Launch your first cluster. Same material, with a transcript.