# Score your own workload first

This course follows one real pipeline: **~7 TB of Sentinel satellite imagery, about 192,000 files.**
Read each file, resample it to a common grid, write the result back. Straightforward work; a great deal
of it.

:::warning Illustrative figures
The 7 TB / 192,000-file / ~3-minute result is Eugo's published benchmark. The per-file timings,
per-stage shares, and scaling numbers in this course are **illustrative**: they show the shape of the
finding rather than measurements from that specific run, and are being confirmed against the original
[profiling](/glossary/profiling) data.

The reasoning is the lesson. Measure your own workload for numbers you will act on.
:::

Before following someone else's pipeline, it is worth finding out whether yours has the same shape.
Have a candidate workload in mind and score it as you go.

## The four properties

Distribution pays when all four hold. Answer each for your own work.

### 1. Are the units independent?

Each file resamples without reference to any other: no [shared state](/glossary/shared-state), no ordering requirement, no
cross-file dependency.

**Your workload:** can unit N be processed without unit N-1's result? If not, more workers cannot help
until the work is restructured. This is the property that disqualifies, so check it first.

### 2. Are there many of them?

192,000 units gives a scheduler plenty to balance.

**Your workload:** count them. Fewer than your worker count means idle workers by definition.

### 3. Is each unit a sensible size?

At roughly a second per file, task bodies sit comfortably above the point where [scheduling overhead](/glossary/scheduling-overhead)
would dominate. Nothing needed batching.

**Your workload:** time one unit. Milliseconds means you will need to batch;
[fan-out](/lesson/distributed-python-with-eugo-hpc/parallel-fan-out) covers how.

### 4. Can every worker reach the data?

The imagery was already in [object storage](/glossary/object-storage), so all workers read in parallel.

**Your workload:** is the input somewhere a cluster can read? If it is on one machine's local disk, the
first job is moving it, before any of this applies.

## Score it

Four yeses means distributing is worth trying. A no on property 1 means restructure before scaling; a no
on 3 means batch first; a no on 4 means move the data first.

The [workload readiness checklist](/resources/workload-readiness-checklist) is the longer form, for when
you are doing this against something real.

## Measure your baseline now

Whatever your workload is, get this number before changing anything:

```python
import time

start = time.perf_counter()
result = process_one_unit(first_unit)     # your actual per-unit work
elapsed = time.perf_counter() - start

print(f"one unit: {elapsed:.2f}s")
print(f"projected serial: {elapsed * total_units / 3600:.1f} hours")
```

Write both numbers down. Every sizing decision in this course depends on the first one, and you cannot
claim an improvement later without the second.

For the satellite pipeline that projection was **around 48 hours**. Two days of wall-clock, with no
partial results until it finished.

## The result, and why the ratio is the least interesting part

End to end on a cluster: **about three minutes.**

Worth qualifying immediately, because a bare "48 hours to 3 minutes" invites the wrong conclusion. The
speedup is not linear, and it is not what any workload gets.

**The finding was that most of those three minutes were I/O, not computation.**

That changed everything downstream: partitioning strategy mattered more than worker count past a
certain point, and adding workers stopped helping well before the budget ran out.

We did not know it at the start. We found it by profiling, which is [the next lesson](./02-profiling-first.mdx)
and the reason this course spends four lessons on measurement and one on the answer.

## What you now know

- The four properties, scored against your own workload rather than read about.
- Your per-unit baseline and serial projection, which every later decision needs.
- That the headline ratio is the least transferable part of any scaling story.

| Next | |
| --- | --- |
| [Profile before you parallelize](./02-profiling-first.mdx) | Where the time actually went |
| [Partitioning 192,000 files](./03-partitioning.mdx) | Why file count is the wrong unit |
| [When I/O is the bottleneck](./04-io-bound-vs-cpu-bound.mdx) | Diagnosing your regime |
| [What we measured](./05-what-we-measured.mdx) | Full numbers, including the surprises |

---

Source: https://university.eugo.io/lesson/scaling-a-real-workload/the-workload