# Profile before you parallelize

:::warning Illustrative figures
The 7 TB / 192,000-file / ~3-minute result is Eugo's published benchmark. The per-file timings,
per-stage shares, and scaling numbers in this lesson are **illustrative**: they show the shape of
the finding rather than measurements from that specific run, and are being confirmed against the
original [profiling](/glossary/profiling) data.

The reasoning is the lesson. Measure your own workload for numbers you will act on.
:::


The step most often skipped, and the one that decides whether the rest of the work pays off.

## Why it matters here

Our assumption was that resampling, the geospatial arithmetic, dominated. Reasonable: it is the part
that sounds expensive.

## Measure a single unit

```python
import time
from contextlib import contextmanager

timings = {}

@contextmanager
def phase(name):
    start = time.perf_counter()
    yield
    timings[name] = timings.get(name, 0) + time.perf_counter() - start

# One representative file, end to end
with phase("read"):
    raster = read_from_object_storage(path)

with phase("decode"):
    array = decode(raster)

with phase("resample"):
    result = resample(array, target_grid)

with phase("encode"):
    encoded = encode(result)

with phase("write"):
    write_to_object_storage(out_path, encoded)

total = sum(timings.values())
for name, seconds in timings.items():
    print(f"{name:10s} {seconds:6.3f}s  {seconds / total:5.1%}")
```

## What we found

```
read        0.412s   47.0%
decode      0.180s   20.5%
resample    0.148s   16.9%
encode      0.096s   11.0%
write       0.041s    4.7%
```

Reading was **47%**. Resampling was **17%**, well under the share we had assumed.

Read plus write is 52% of the time. Just over half of this workload is I/O.

## Why that changes the plan

Amdahl's law: optimizing a stage caps your gain at that stage's share of runtime.

Making resampling infinitely fast would have saved 17%. Weeks of numerical optimization for a sixth of
the runtime.

More importantly, a workload where half the time is I/O behaves differently under scaling. [Object storage](/glossary/object-storage)
is a **shared resource**. Every worker reads through the same pipe, so doubling workers does not double
read bandwidth, so [speedup](/glossary/speedup) flattens early. See
[when I/O is the bottleneck](./04-io-bound-vs-cpu-bound.mdx).

Knowing this before provisioning meant not paying for 64 workers to discover that 16 was the useful
ceiling.

## Profile at a realistic size

One caveat: a small sample can mislead.

Overheads that dominate at small scale vanish at large. Memory pressure invisible at small scale
dominates at large. Caching effects on a warm file do not represent a cold read.

Use a size where the real bottleneck is present. For us, a handful of representative files at full
resolution, not a downsampled thumbnail.

## Check variance

Profile several files, not one:

```python
samples = [profile_one(p) for p in random.sample(paths, 20)]
for stage in samples[0]:
    values = sorted(s[stage] for s in samples)
    print(f"{stage:10s} median {values[len(values)//2]:.3f}s  max {values[-1]:.3f}s")
```

Our read times ranged from 0.2s to 1.4s: a 7x spread, because files varied in size. That spread is the
[long tail](./03-partitioning.mdx) waiting to happen, and it is invisible if you profile one file.

## The general lesson

Before distributing anything, know:

1. **The share each stage takes.** It caps what optimizing that stage can win.
2. **The I/O-to-compute ratio**, which predicts how well the job will scale.
3. **The variance across units.** A wide spread means a [long tail](/glossary/long-tail).

Twenty minutes of profiling regularly redirects days of work. The
[workload readiness checklist](/resources/workload-readiness-checklist) is the short form.

---

Source: https://university.eugo.io/lesson/scaling-a-real-workload/profiling-first