Score your own workload first
This course follows one real pipeline: ~7 TB of Sentinel satellite imagery, about 192,000 files. Read each file, resample it to a common grid, write the result back. Straightforward work; a great deal of it.
The 7 TB / 192,000-file / ~3-minute result is Eugo's published benchmark. The per-file timings, per-stage shares, and scaling numbers in this course are illustrative: they show the shape of the finding rather than measurements from that specific run, and are being confirmed against the original profiling data.
The reasoning is the lesson. Measure your own workload for numbers you will act on.
Before following someone else's pipeline, it is worth finding out whether yours has the same shape. Have a candidate workload in mind and score it as you go.
The four properties
Distribution pays when all four hold. Answer each for your own work.
1. Are the units independent?
Each file resamples without reference to any other: no shared state, no ordering requirement, no cross-file dependency.
Your workload: can unit N be processed without unit N-1's result? If not, more workers cannot help until the work is restructured. This is the property that disqualifies, so check it first.
2. Are there many of them?
192,000 units gives a scheduler plenty to balance.
Your workload: count them. Fewer than your worker count means idle workers by definition.
3. Is each unit a sensible size?
At roughly a second per file, task bodies sit comfortably above the point where scheduling overhead would dominate. Nothing needed batching.
Your workload: time one unit. Milliseconds means you will need to batch; fan-out covers how.
4. Can every worker reach the data?
The imagery was already in object storage, so all workers read in parallel.
Your workload: is the input somewhere a cluster can read? If it is on one machine's local disk, the first job is moving it, before any of this applies.
Score it
Four yeses means distributing is worth trying. A no on property 1 means restructure before scaling; a no on 3 means batch first; a no on 4 means move the data first.
The workload readiness checklist is the longer form, for when you are doing this against something real.
Measure your baseline now
Whatever your workload is, get this number before changing anything:
import time
start = time.perf_counter()
result = process_one_unit(first_unit) # your actual per-unit work
elapsed = time.perf_counter() - start
print(f"one unit: {elapsed:.2f}s")
print(f"projected serial: {elapsed * total_units / 3600:.1f} hours")
Write both numbers down. Every sizing decision in this course depends on the first one, and you cannot claim an improvement later without the second.
For the satellite pipeline that projection was around 48 hours. Two days of wall-clock, with no partial results until it finished.
The result, and why the ratio is the least interesting part
End to end on a cluster: about three minutes.
Worth qualifying immediately, because a bare "48 hours to 3 minutes" invites the wrong conclusion. The speedup is not linear, and it is not what any workload gets.
The finding was that most of those three minutes were I/O, not computation.
That changed everything downstream: partitioning strategy mattered more than worker count past a certain point, and adding workers stopped helping well before the budget ran out.
We did not know it at the start. We found it by profiling, which is the next lesson and the reason this course spends four lessons on measurement and one on the answer.
What you now know
- The four properties, scored against your own workload rather than read about.
- Your per-unit baseline and serial projection, which every later decision needs.
- That the headline ratio is the least transferable part of any scaling story.
| Next | |
|---|---|
| Profile before you parallelize | Where the time actually went |
| Partitioning 192,000 files | Why file count is the wrong unit |
| When I/O is the bottleneck | Diagnosing your regime |
| What we measured | Full numbers, including the surprises |