Skip to main content

Feel the problem for yourself

Arguments about why a platform is worth using are cheap. Ten minutes of measuring is not.

You run the same work twice: once the way you already would, once distributed. Then you compare the two numbers. Everything the rest of the course teaches follows from what you see here.

Set up​

Open a session and a fresh notebook. Paste this:

import time
import numpy as np

def process(seed):
    """Stand-in for a real per-item workload: some numeric work, a few hundred ms of it."""
    rng = np.random.default_rng(seed)
    a = rng.random((900, 900))
    return float((a @ a.T).trace())

ITEMS = list(range(200))

Nothing distributed yet. process is deliberately ordinary, the kind of function that shows up in any analysis, doing real arithmetic over a real array.

Run it the way you would today​

start = time.perf_counter()
serial = [process(i) for i in ITEMS]
print(f"{len(serial)} items in {time.perf_counter() - start:.1f}s")

Let it finish. Write the number down.

On a single machine this takes on the order of a minute or two: 200 items at a few hundred milliseconds each. The exact figure depends on your session size, and that is deliberate. It is your baseline, not a number from a benchmark table.

What you are measuring

One core, one item at a time. The 199 items not currently being processed are waiting.

Now distribute it​

Launch a cluster from the EugoHPC Manager panel with 16 workers, which is a good size for this. Wait for it to report ready.

Then change three lines:

import eugo.hpc

@eugo.hpc.distribute          # 1. the function becomes a task
def process(seed):
    rng = np.random.default_rng(seed)
    a = rng.random((900, 900))
    return float((a @ a.T).trace())

start = time.perf_counter()
futures = [process(i) for i in ITEMS]      # 2. calls return immediately
results = eugo.hpc.get(futures)            # 3. one wait, at the end
print(f"{len(results)} items in {time.perf_counter() - start:.1f}s")

Run it. Write the second number down.

What just happened​

The body of process did not change. Same NumPy, same arithmetic. If you diff the two versions, the function is byte-identical apart from the decorator above it.

The call site barely changed. process(i) still looks like calling a function. What changed is what it returns. Instead of a value you get a future, a handle to a result that does not exist yet. That is why the list comprehension finishes almost instantly while the work is still running.

One wait, not two hundred. eugo.hpc.get() on the whole list blocks once, after every task is already in flight.

The number you should not expect​

Your second measurement will be meaningfully faster, and nowhere near 16×.

That gap matters more than either number on its own. It comes from real costs:

  • Dispatching 200 tasks is not free.
  • The cluster had to hand each worker its arguments and collect each result.
  • The last few tasks finish while most workers are already idle.

A platform that claimed 16× would be lying to you. Knowing the shape of the gap is what lets you size a cluster instead of guessing. Parallel fan-out is where you learn to close it.

Try breaking it​

Worth two minutes, because it is the mistake that costs people the most:

# Move get() inside the loop
start = time.perf_counter()
results = [eugo.hpc.get(process(i)) for i in ITEMS]
print(f"{len(results)} items in {time.perf_counter() - start:.1f}s")

This is slower than the serial version. You now pay dispatch overhead on every item while still running them one at a time. Nothing errors. Nothing warns you.

That is why futures and get() gets a lesson of its own.

Shut the cluster down​

From the same panel you launched it. A cluster bills for as long as it runs, whether or not it is doing anything. That is the most common avoidable cost on the platform.

What you now know​

  • Distributing existing Python is a decorator and one get(), not a rewrite.
  • A distributed call returns a future, not a value.
  • Real speedup falls short of worker count, for reasons you can name.
  • get() in the wrong place makes things worse, silently.

Next: open your first interactive session properly, and find out which of your files survive a restart.