# Measuring the gain honestly

It is easy to produce a number that flatters the change you just made. This lesson is about the traps,
because a measurement you cannot trust is worse than none. It makes you confidently wrong.

## Always have a baseline

Record the timing before the change:

```python
import time

start = time.perf_counter()
result = original_version(data)
baseline = time.perf_counter() - start
print(f"baseline {baseline:.3f}s")
```

Without it, "faster" is an impression.

## Trap 1: measuring once

Timings vary with caches, other tenants, and scheduling. One measurement of each version tells you
little.

```python
def median_time(fn, arg, runs=5):
    times = []
    for _ in range(runs):
        start = time.perf_counter()
        fn(arg)
        times.append(time.perf_counter() - start)
    return sorted(times)[len(times) // 2]
```

Median rather than mean, so one outlier from an unrelated hiccup does not move the answer.

## Trap 2: the first run

First runs include one-time costs: imports, JIT warm-up, device initialization, cold caches. Discard
one run before measuring.

```python
fn(arg)                       # warm up, discard
timing = median_time(fn, arg)
```

## Trap 3: not synchronizing GPU work

The big one. CUDA operations queue asynchronously, so timing around them measures submission, not
completion:

```python
# WRONG: measures how fast you queued work
start = time.perf_counter()
result = gpu_work(tensor)
elapsed = time.perf_counter() - start      # implausibly small

# RIGHT
start = time.perf_counter()
result = gpu_work(tensor)
torch.cuda.synchronize()
elapsed = time.perf_counter() - start
```

If a [GPU](/glossary/gpu) [speedup](/glossary/speedup) looks too good to be true, check this first. It usually is.

## Trap 4: measuring a different problem

Comparisons must do identical work. Common ways this slips:

- The optimized version reads cached data; the baseline read from storage
- Different input sizes
- The optimized version returns a summary; the baseline returned full rows
- Different `dtype`. `float32` versus `float64` is not the same problem

## Trap 5: extrapolating from a small sample

A 100-row sample says little about 100 million rows. Overheads that dominate at small scale vanish at
large; memory pressure that is invisible at small scale dominates at large.

Test at a size where the real bottleneck is present, even if not at full scale.

## Trap 6: ignoring where the time actually is

Amdahl's law: if the part you optimized is 20% of runtime, making it infinitely fast gains you 20%.

```python
# Measure the whole and the part
total = median_time(full_pipeline, data)
part = median_time(the_bit_i_optimized, data)
print(f"optimizing this can save at most {part / total:.0%}")
```

Run this *before* optimizing. It regularly redirects effort to somewhere it pays.

## Trap 7: not reporting cost

A run twice as fast on four times the [workers](/glossary/worker) is not obviously a win.

| | Runtime | Workers | Worker-seconds |
| --- | --- | --- | --- |
| Before | 40 min | 8 | 19,200 |
| After | 20 min | 32 | 38,400 |

Twice as fast, twice the cost. Sometimes that is exactly the trade you want; it should be a decision
rather than a surprise. See the
[cost optimization checklist](/resources/cost-optimization-checklist).

## A template worth reusing

```python
def compare(baseline_fn, optimized_fn, data, runs=5):
    baseline_fn(data); optimized_fn(data)          # warm up
    b = median_time(baseline_fn, data, runs)
    o = median_time(optimized_fn, data, runs)
    print(f"baseline  {b:.3f}s")
    print(f"optimized {o:.3f}s")
    print(f"speedup   {b / o:.2f}x")
    return b / o
```

## What to write down

| | Value |
| --- | --- |
| Input size measured | |
| Baseline (median of 5) | |
| Optimized (median of 5) | |
| Speedup | |
| Worker-seconds before / after | |
| Share of total runtime this stage represents | |

That last row is the one that stops a 30x speedup on 2% of the runtime from being reported as a 30x
speedup.

---

Source: https://university.eugo.io/lesson/how-automatic-optimizations-work/measuring-the-gain