Skip to main content

Watching a run and reading the logs

A distributed run that is working and one that is stuck can look identical from the notebook. The cell just sits there either way. Knowing where to look is the difference between a minute and an afternoon.

What a healthy run looks like​

  • Tasks distributed across workers, not concentrated on one
  • Task completions accumulating steadily
  • Workers busy rather than idle

What a stuck run looks like​

  • Tasks submitted but none started. Usually a resource request nothing satisfies.
  • All work on one worker. Same cause, commonly a GPU request.
  • Steady progress that then stops with a few tasks outstanding. That is a long tail, where one oversized partition holds up completion.

Errors inside tasks​

An exception in a task body does not surface until you resolve the future:

futures = [risky.remote(x) for x in items]
results = eugo.hpc.get(futures) # the exception is raised here

The traceback comes from inside the task, which is what you want, but the line number refers to the task body, not the get() call. Read past the first frame.

Finding which input failed​

When one task in a thousand fails, the traceback alone will not tell you which input caused it. Catch and return the failure instead of raising:

@eugo.hpc.distribute
def safe_process(item):
try:
return {"item": item, "ok": True, "result": process_item(item)} # a plain function
except Exception as exc:
return {"item": item, "ok": False, "error": repr(exc)}

results = eugo.hpc.get([safe_process.remote(i) for i in items])
failures = [r for r in results if not r["ok"]]
print(f"{len(failures)} failed")
for f in failures[:5]:
print(f["item"], f["error"])

This turns "something threw" into "these seventeen inputs threw, and here is why." On a long run it is worth the extra few lines, because the alternative is re-running everything to find out.

Printing from inside a task​

print() inside a task body shows up in your notebook, prefixed with the worker that printed it, and it is also kept in that worker's log. Similar lines from different workers may be folded into one, with a note saying how many times it repeated. Returning diagnostic information, as above, is still more reliable: printed lines from many tasks arrive mixed together, sometimes after the cell has moved on, while a returned value stays with the input it describes.

Timing a run​

import time

start = time.perf_counter()
results = eugo.hpc.get([process.remote(p) for p in partitions])
print(f"{len(results)} tasks in {time.perf_counter() - start:.1f}s")

Record this number. Without a baseline, "it got faster" is an impression rather than a measurement. Measuring the gain honestly is its own skill.

When a run is slower than expected​

Work through the performance tuning checklist. The first item, a get() call inside a loop, accounts for more missing speedup than everything else combined.