Skip to main content

Profile before you parallelize

Illustrative figures

The 7 TB / 192,000-file / ~3-minute result is Eugo's published benchmark. The per-file timings, per-stage shares, and scaling numbers in this lesson are illustrative: they show the shape of the finding rather than measurements from that specific run, and are being confirmed against the original profiling data.

The reasoning is the lesson. Measure your own workload for numbers you will act on.

The step most often skipped, and the one that decides whether the rest of the work pays off.

Why it matters here​

Our assumption was that resampling, the geospatial arithmetic, dominated. Reasonable: it is the part that sounds expensive.

Measure a single unit​

import time
from contextlib import contextmanager

timings = {}

@contextmanager
def phase(name):
    start = time.perf_counter()
    yield
    timings[name] = timings.get(name, 0) + time.perf_counter() - start

# One representative file, end to end
with phase("read"):
    raster = read_from_object_storage(path)

with phase("decode"):
    array = decode(raster)

with phase("resample"):
    result = resample(array, target_grid)

with phase("encode"):
    encoded = encode(result)

with phase("write"):
    write_to_object_storage(out_path, encoded)

total = sum(timings.values())
for name, seconds in timings.items():
    print(f"{name:10s} {seconds:6.3f}s  {seconds / total:5.1%}")

What we found​

read        0.412s   47.0%
decode      0.180s   20.5%
resample    0.148s   16.9%
encode      0.096s   11.0%
write       0.041s    4.7%

Reading was 47%. Resampling was 17%, well under the share we had assumed.

Read plus write is 52% of the time. Just over half of this workload is I/O.

Why that changes the plan​

Amdahl's law: optimizing a stage caps your gain at that stage's share of runtime.

Making resampling infinitely fast would have saved 17%. Weeks of numerical optimization for a sixth of the runtime.

More importantly, a workload where half the time is I/O behaves differently under scaling. Object storage is a shared resource. Every worker reads through the same pipe, so doubling workers does not double read bandwidth, so speedup flattens early. See when I/O is the bottleneck.

Knowing this before provisioning meant not paying for 64 workers to discover that 16 was the useful ceiling.

Profile at a realistic size​

One caveat: a small sample can mislead.

Overheads that dominate at small scale vanish at large. Memory pressure invisible at small scale dominates at large. Caching effects on a warm file do not represent a cold read.

Use a size where the real bottleneck is present. For us, a handful of representative files at full resolution, not a downsampled thumbnail.

Check variance​

Profile several files, not one:

samples = [profile_one(p) for p in random.sample(paths, 20)]
for stage in samples[0]:
    values = sorted(s[stage] for s in samples)
    print(f"{stage:10s} median {values[len(values)//2]:.3f}s  max {values[-1]:.3f}s")

Our read times ranged from 0.2s to 1.4s: a 7x spread, because files varied in size. That spread is the long tail waiting to happen, and it is invisible if you profile one file.

The general lesson​

Before distributing anything, know:

  1. The share each stage takes. It caps what optimizing that stage can win.
  2. The I/O-to-compute ratio, which predicts how well the job will scale.
  3. The variance across units. A wide spread means a long tail.

Twenty minutes of profiling regularly redirects days of work. The workload readiness checklist is the short form.