Measuring the gain honestly

It is easy to produce a number that flatters the change you just made. This lesson is about the traps, because a measurement you cannot trust is worse than none. It makes you confidently wrong.

Always have a baseline

Record the timing before the change:

import time

start = time.perf_counter()
result = original_version(data)
baseline = time.perf_counter() - start
print(f"baseline {baseline:.3f}s")

Without it, "faster" is an impression.

Trap 1: measuring once

Timings vary with caches, other tenants, and scheduling. One measurement of each version tells you little.

def median_time(fn, arg, runs=5):
    times = []
    for _ in range(runs):
        start = time.perf_counter()
        fn(arg)
        times.append(time.perf_counter() - start)
    return sorted(times)[len(times) // 2]

Median rather than mean, so one outlier from an unrelated hiccup does not move the answer.

Trap 2: the first run

First runs include one-time costs: imports, JIT warm-up, device initialization, cold caches. Discard one run before measuring.

fn(arg)                       # warm up, discard
timing = median_time(fn, arg)

Trap 3: not synchronizing GPU work

The big one. CUDA operations queue asynchronously, so timing around them measures submission, not completion:

# WRONG: measures how fast you queued work
start = time.perf_counter()
result = gpu_work(tensor)
elapsed = time.perf_counter() - start      # implausibly small

# RIGHT
start = time.perf_counter()
result = gpu_work(tensor)
torch.cuda.synchronize()
elapsed = time.perf_counter() - start

If a GPU speedup looks too good to be true, check this first. It usually is.

Trap 4: measuring a different problem

Comparisons must do identical work. Common ways this slips:

• The optimized version reads cached data; the baseline read from storage
• Different input sizes
• The optimized version returns a summary; the baseline returned full rows
• Different dtype. float32 versus float64 is not the same problem

Trap 5: extrapolating from a small sample

A 100-row sample says little about 100 million rows. Overheads that dominate at small scale vanish at large; memory pressure that is invisible at small scale dominates at large.

Test at a size where the real bottleneck is present, even if not at full scale.

Trap 6: ignoring where the time actually is

Amdahl's law: if the part you optimized is 20% of runtime, making it infinitely fast gains you 20%.

# Measure the whole and the part
total = median_time(full_pipeline, data)
part = median_time(the_bit_i_optimized, data)
print(f"optimizing this can save at most {part / total:.0%}")

Run this before optimizing. It regularly redirects effort to somewhere it pays.

Trap 7: not reporting cost

A run twice as fast on four times the workers is not obviously a win.

	Runtime	Workers	Worker-seconds
Before	40 min	8	19,200
After	20 min	32	38,400

Twice as fast, twice the cost. Sometimes that is exactly the trade you want; it should be a decision rather than a surprise. See the cost optimization checklist.

A template worth reusing

def compare(baseline_fn, optimized_fn, data, runs=5):
    baseline_fn(data); optimized_fn(data)          # warm up
    b = median_time(baseline_fn, data, runs)
    o = median_time(optimized_fn, data, runs)
    print(f"baseline  {b:.3f}s")
    print(f"optimized {o:.3f}s")
    print(f"speedup   {b / o:.2f}x")
    return b / o

What to write down

	Value
Input size measured
Baseline (median of 5)
Optimized (median of 5)
Speedup
Worker-seconds before / after
Share of total runtime this stage represents

That last row is the one that stops a 30x speedup on 2% of the runtime from being reported as a 30x speedup.