Skip to main content

Watching a run and reading the logs

A distributed run that is working and one that is stuck can look identical from the notebook. The cell just sits there either way. Knowing where to look is the difference between a minute and an afternoon.

What a healthy run looks like​

  • Tasks distributed across workers, not concentrated on one
  • Task completions accumulating steadily
  • Workers busy rather than idle

What a stuck run looks like​

  • Tasks submitted but none started. Usually a resource request nothing satisfies.
  • All work on one worker. Same cause, commonly a GPU request.
  • Steady progress that then stops with a few tasks outstanding. That is a long tail, where one oversized partition holds up completion.

Errors inside tasks​

An exception in a task body does not surface until you resolve the future:

futures = [risky(x) for x in items]
results = eugo.hpc.get(futures)   # the exception is raised here

The traceback comes from inside the task, which is what you want, but the line number refers to the task body, not the get() call. Read past the first frame.

Finding which input failed​

When one task in a thousand fails, the traceback alone will not tell you which input caused it. Catch and return the failure instead of raising:

@eugo.hpc.distribute
def safe_process(item):
    try:
        return {"item": item, "ok": True, "result": process(item)}
    except Exception as exc:
        return {"item": item, "ok": False, "error": repr(exc)}

results = eugo.hpc.get([safe_process(i) for i in items])
failures = [r for r in results if not r["ok"]]
print(f"{len(failures)} failed")
for f in failures[:5]:
    print(f["item"], f["error"])

This turns "something threw" into "these seventeen inputs threw, and here is why." On a long run it is worth the extra few lines, because the alternative is re-running everything to find out.

Printing from inside a task​

print() inside a task body writes to that worker's output, not your notebook. Returning diagnostic information, as above, is more reliable than hunting through worker logs.

Timing a run​

import time

start = time.perf_counter()
results = eugo.hpc.get([process(p) for p in partitions])
print(f"{len(results)} tasks in {time.perf_counter() - start:.1f}s")

Record this number. Without a baseline, "it got faster" is an impression rather than a measurement. Measuring the gain honestly is its own skill.

When a run is slower than expected​

Work through the performance tuning checklist. The first item, a get() call inside a loop, accounts for more missing speedup than everything else combined.