Skip to main content

What we measured, and what surprised us

Figures pending verification

The headline result below is Eugo's published benchmark. The per-stage shares and the scaling curve are illustrative: they show the shape of the finding, not measurements from this specific run, and the exact numbers are being confirmed against the original profiling data.

Treat the reasoning as the lesson and the numbers as provisional. Where a figure matters to a decision you are making, measure your own workload. That is what the whole course argues.

The headline​

Input~7 TB, ~192,000 Sentinel files
Workread, geospatially resample, write back to object storage
End-to-end runtime~3 minutes

That number is real and it is the reason this course exists. It is also the least interesting thing here, because it tells you nothing you can apply to your own work.

The finding that actually mattered​

Most of that time was I/O, not computation.

Profiling a single file put read and write together at roughly half the per-unit time. The geospatial resampling, the part that sounds expensive, came in well under a quarter.

That one ratio changed every subsequent decision:

  • Optimizing the resampling math would have been capped at whatever share it holds. Weeks of numerical work for a fraction of the runtime.
  • Object storage is a shared resource. Every worker reads through the same aggregate bandwidth, so the speedup curve flattens far earlier than a compute-bound job's would.
  • Partitioning strategy mattered more than worker count past the flattening point.

None of that was visible before profiling. All of it was obvious after.

Three things that surprised us​

1. The expensive-sounding stage was not the bottleneck​

We expected the geospatial arithmetic to dominate. Reading did. We had been ready to optimize the wrong stage, and only profiling stopped us.

2. More workers stopped helping well before the plan's limit​

Past the point where object-storage bandwidth saturated, each added worker got a thinner slice of the same pipe. Runtime barely moved while cost rose in proportion to worker count.

The instinct is to use whatever the plan allows. The curve says otherwise. The difference between "fastest" and "best value" was several multiples of spend for a few percent of runtime.

3. File-size variance mattered more than file count​

The files varied substantially in size. Partitioning by count therefore produced a long tail: most workers finished early and idled while a handful ground through oversized partitions.

Switching to weight-based batching, largest-first, cut wall-clock time at the same worker count. Free speedup from partitioning arithmetic rather than hardware. See partitioning.

What we would do differently​

Profile the variance as well as the median. We profiled one file, got the stage shares roughly right, and missed the size spread entirely. Twenty files would have shown it immediately.

Find the flattening point on a subset first. We provisioned before establishing where gains stopped. A small sample run at 4, 8, 16, and 32 workers would have identified the right size in minutes.

Instrument from the start. Per-stage timing was added after the first full run raised questions. Having it from the beginning would have answered them without a re-run.

What transfers to your own workload​

This is the part worth keeping, independent of any number above.

Profile before parallelizing. The bottleneck is regularly not where you assume, and Amdahl's law caps what optimizing the wrong stage can win.

Establish your regime. Compute-bound scales with workers; I/O-bound flattens early. That single distinction decides whether more workers is the answer or the mistake. See when I/O is the bottleneck.

Partition by cost, not count. Whenever units vary in size, which with files is almost always.

Find where the curve bends. The right worker count is where gains stop, not the plan maximum. Measure it on a subset before committing.

Report runtime and cost together. "Twice as fast" and "twice the worker-hours" are both true of the same change. Reporting only the first is not a lie, but it is not a decision anyone can act on.

Next​

Video: Seven terabytes in three minutes, the same material with a transcript.