# Watching a run and reading the logs

A distributed run that is working and one that is stuck can look identical from the [notebook](/glossary/notebook). The
cell just sits there either way. Knowing where to look is the difference between a minute and an afternoon.

## What a healthy run looks like

- Tasks distributed across workers, not concentrated on one
- Task completions accumulating steadily
- Workers busy rather than idle

## What a stuck run looks like

- Tasks submitted but none started. Usually a resource request nothing satisfies.
- All work on one worker. Same cause, commonly a GPU request.
- Steady progress that then stops with a few tasks outstanding. That is a **long tail**, where one
  oversized [partition](/glossary/partition) holds up completion.

## Errors inside tasks

An exception in a task body does not surface until you resolve the [future](/glossary/future):

```python
futures = [risky(x) for x in items]
results = eugo.hpc.get(futures)   # the exception is raised here
```

The traceback comes from inside the task, which is what you want, but the line number refers to the
task body, not the `get()` call. Read past the first frame.

## Finding which input failed

When one task in a thousand fails, the traceback alone will not tell you which input caused it. Catch
and return the failure instead of raising:

```python
@eugo.hpc.distribute
def safe_process(item):
    try:
        return {"item": item, "ok": True, "result": process(item)}
    except Exception as exc:
        return {"item": item, "ok": False, "error": repr(exc)}

results = eugo.hpc.get([safe_process(i) for i in items])
failures = [r for r in results if not r["ok"]]
print(f"{len(failures)} failed")
for f in failures[:5]:
    print(f["item"], f["error"])
```

This turns "something threw" into "these seventeen inputs threw, and here is why." On a long run it is
worth the extra few lines, because the alternative is re-running everything to find out.

## Printing from inside a task

`print()` inside a task body writes to that worker's output, not your notebook. Returning diagnostic
information, as above, is more reliable than hunting through worker logs.

## Timing a run

```python
import time

start = time.perf_counter()
results = eugo.hpc.get([process(p) for p in partitions])
print(f"{len(results)} tasks in {time.perf_counter() - start:.1f}s")
```

Record this number. Without a baseline, "it got faster" is an impression rather than a measurement.
[Measuring the gain honestly](/courses/how-automatic-optimizations-work) is its own skill.

## When a run is slower than expected

Work through the
[performance tuning checklist](/resources/performance-tuning-checklist). The first item, a `get()`
call inside a loop, accounts for more missing [speedup](/glossary/speedup) than everything else combined.

---

Source: https://university.eugo.io/lesson/eugoide-essentials/monitoring-a-run