Parallel fan-out over many inputs
Fan-out is launching many independent tasks at once and collecting results together. It is the shape most Eugo code takes.
futures = [process(item) for item in items]
results = eugo.hpc.get(futures)
The mechanics are two lines. The judgment is in partition size and expectations.
Partition size
Too small, and scheduling overhead dominates:
# 1M tasks doing microseconds of work each. Overhead swamps everything.
futures = [add_one(x) for x in range(1_000_000)]
Too large, and you lose parallelism at the end. With 4 partitions and 32 workers, 28 sit idle immediately.
Aim for task bodies measured in seconds, and for meaningfully more partitions than workers so the scheduler can keep everyone busy:
def chunks(seq, size):
for i in range(0, len(seq), size):
yield seq[i:i + size]
@eugo.hpc.distribute
def process_batch(batch):
return [expensive(x) for x in batch]
futures = [process_batch(c) for c in chunks(items, 500)]
results = [r for batch in eugo.hpc.get(futures) for r in batch]
The long tail
Uneven partitions produce a characteristic failure: progress races along, then stalls with a few tasks outstanding while nearly every worker sits idle.
The run finishes when the slowest task finishes. One partition ten times larger than the others sets your wall-clock time on its own.
Adding workers does not help. The oversized partition is already on one worker and cannot be split across more. Split it in the data instead:
# Size partitions by weight rather than count
def weighted_chunks(items, target_weight):
batch, weight = [], 0
for item in items:
batch.append(item)
weight += estimate_cost(item)
if weight >= target_weight:
yield batch
batch, weight = [], 0
if batch:
yield batch
For file processing, file size is usually a good enough cost proxy.
What speedup to expect
Ideal speedup equals worker count. Real speedup never reaches it.
The speedup figures below are illustrative — no capture backs them. They show the shape of how each regime scales, not measurements of Eugo hardware. Do not quote them as Eugo performance figures; measure your own workload.
| Workload | Speedup at 16 workers |
|---|---|
| Compute-bound, even partitions | ~13x |
| Mixed compute and I/O | ~9x |
| I/O-bound | ~4x |
Those are illustrative figures, not measurements. The full curves are in the speedup explorer, which says the same on its own face.
The I/O-bound row is the important one. If most of your time is spent waiting on object storage, extra workers add waiting rather than throughput. Past roughly 8 workers such a workload barely improves, and past 16 it flatlines while continuing to cost more.
Establish which regime you are in before scaling. See when I/O is the bottleneck.
Handling failures across many tasks
One get() on a thousand futures raises on the first failure and tells you nothing about which input
caused it. Return failures instead of raising:
@eugo.hpc.distribute
def safe(item):
try:
return {"item": item, "ok": True, "value": process(item)}
except Exception as exc:
return {"item": item, "ok": False, "error": repr(exc)}
results = eugo.hpc.get([safe(i) for i in items])
ok = [r["value"] for r in results if r["ok"]]
bad = [r for r in results if not r["ok"]]
print(f"{len(ok)} succeeded, {len(bad)} failed")
On a long run this is worth the extra lines. The alternative is re-running everything to discover which input was malformed.
A checklist before you scale
- Are the units genuinely independent?
- Are partitions roughly even in cost?
- Are task bodies seconds rather than milliseconds?
- Is the work compute-bound or I/O-bound?
- Do you have a baseline timing to compare against?
The workload readiness checklist is the longer form.