# What we measured, and what surprised us

:::warning Figures pending verification
The headline result below is Eugo's published benchmark. The **per-stage shares and the scaling
curve are illustrative**: they show the shape of the finding, not measurements from this specific
run, and the exact numbers are being confirmed against the original profiling data.

Treat the *reasoning* as the lesson and the *numbers* as provisional. Where a figure matters to a
decision you are making, measure your own workload. That is what the whole course argues.
:::

## The headline

| | |
| --- | --- |
| Input | ~7 TB, ~192,000 Sentinel files |
| Work | read, geospatially resample, write back to object storage |
| End-to-end runtime | **~3 minutes** |

That number is real and it is the reason this course exists. It is also the least interesting thing
here, because it tells you nothing you can apply to your own work.

## The finding that actually mattered

**Most of that time was I/O, not computation.**

Profiling a single file put read and write together at roughly half the per-unit time. The geospatial
resampling, the part that *sounds* expensive, came in well under a quarter.

That one ratio changed every subsequent decision:

- Optimizing the resampling math would have been capped at whatever share it holds. Weeks of
  numerical work for a fraction of the runtime.
- [Object storage](/glossary/object-storage) is a **shared** resource. Every worker reads through the same aggregate bandwidth,
  so the speedup curve flattens far earlier than a [compute-bound](/glossary/compute-bound) job's would.
- Partitioning strategy mattered more than worker count past the flattening point.

None of that was visible before profiling. All of it was obvious after.

## Three things that surprised us

### 1. The expensive-sounding stage was not the bottleneck

We expected the geospatial arithmetic to dominate. Reading did. We had been ready to optimize the
wrong stage, and only profiling stopped us.

### 2. More workers stopped helping well before the plan's limit

Past the point where object-storage bandwidth saturated, each added worker got a thinner slice of
the same pipe. Runtime barely moved while cost rose in proportion to worker count.

The instinct is to use whatever the plan allows. The curve says otherwise. The difference between
"fastest" and "best value" was several multiples of spend for a few percent of runtime.

### 3. File-size variance mattered more than file count

The files varied substantially in size. Partitioning by *count* therefore produced a [long tail](/glossary/long-tail):
most workers finished early and idled while a handful ground through oversized partitions.

Switching to weight-based batching, largest-first, cut wall-clock time at the same worker count.
Free speedup from partitioning arithmetic rather than hardware. See
[partitioning](./03-partitioning.mdx).

## What we would do differently

**Profile the variance as well as the median.** We profiled one file, got the stage shares roughly
right, and missed the size spread entirely. Twenty files would have shown it immediately.

**Find the flattening point on a subset first.** We provisioned before establishing where gains
stopped. A small sample run at 4, 8, 16, and 32 workers would have identified the right size in
minutes.

**Instrument from the start.** Per-stage timing was added after the first full run raised questions.
Having it from the beginning would have answered them without a re-run.

## What transfers to your own workload

This is the part worth keeping, independent of any number above.

**Profile before parallelizing.** The bottleneck is regularly not where you assume, and Amdahl's law
caps what optimizing the wrong stage can win.

**Establish your regime.** Compute-bound scales with workers; [I/O-bound](/glossary/io-bound) flattens early. That single
distinction decides whether more workers is the answer or the mistake. See
[when I/O is the bottleneck](./04-io-bound-vs-cpu-bound.mdx).

**Partition by cost, not count.** Whenever units vary in size, which with files is almost always.

**Find where the curve bends.** The right worker count is where gains stop, not the plan maximum.
Measure it on a subset before committing.

**Report runtime and cost together.** "Twice as fast" and "twice the worker-hours" are both true of
the same change. Reporting only the first is not a lie, but it is not a decision anyone can act on.

## Next

- [Performance tuning checklist](/resources/performance-tuning-checklist), the working form of this
- [Cluster sizer](/interactive), to estimate the bend point for your own workload

---

**Video:** [Seven terabytes in three minutes](/videos/seven-terabytes-in-three-minutes), the same material with a transcript.

---

Source: https://university.eugo.io/lesson/scaling-a-real-workload/what-we-measured