Python Benchmarking

Wrapping code between two time.time() calls feels like measuring performance, and in a narrow, technically-true sense it is — but a single measurement captures not just your code's execution time, but also whatever OS scheduling happened to occur during that exact window, whatever CPU cache state happened to exist, and whatever else was running on the machine competing for resources at that moment. Run the identical code five times and you'll typically see five different numbers, sometimes varying by a meaningful percentage — and from a single run, there's no way to know whether a given number reflects the code's true typical speed or an unusually fast or slow moment.

This matters enormously when comparing two implementations: if implementation A's single measurement happens to land on a naturally faster moment and implementation B's happens to land on a slower one, you can easily conclude 'A is faster' when the two are actually comparable, or even when B is genuinely faster in typical conditions. This is precisely the trap that makes ad-hoc time.time() comparisons a common source of incorrect performance conclusions, even among engineers who know better in principle.

The fix isn't a smarter single measurement — it's *repetition*: running the same code many times and looking at the aggregate (average, or better, median/minimum) rather than trusting any one run in isolation, which is exactly what a dedicated benchmarking tool automates for you.

timeit.timeit(statement, setup=..., number=1000) runs the given statement the specified number of times (1000, here) in a tightly controlled loop, and returns the *total* time for all those runs — dividing by number gives you the average per-call time, a number that's dramatically more stable and reproducible than any single measurement, since random noise sources tend to average out across a large number of repetitions.

The setup parameter matters specifically because it lets you exclude one-time setup cost (building large_list, in the example) from the timed portion — you want to measure sorted(large_list)'s cost specifically, not sorted(large_list) plus the cost of constructing large_list in the first place, which would happen only once if written naively but would otherwise contaminate every one of the 1000 timed iterations if included inside the timed statement itself.

timeit also deliberately disables Python's automatic garbage collection during its timed runs by default — garbage collection can trigger unpredictably and add noise-like variance to timing measurements, so timeit controls for it specifically to produce a more consistent, comparable number across repeated runs (and across separate timeit.timeit() calls being compared against each other).

Benchmarking two implementations meaningfully against each other requires holding every variable *except* the code being compared constant: the same input data (data = list(range(10_000)), shared identically via setup for both timed statements), the same number of repetitions, run on the same machine in roughly the same conditions (ideally back-to-back, minimizing the chance that unrelated system load shifted meaningfully between the two benchmark runs).

The list-comprehension-versus-map() comparison in the scene is a genuinely common real question professional Python engineers have opinions about, and it's precisely the kind of question that should be settled by measurement, not intuition or a blog post's claim from a different Python version — timeit run identically for both candidates gives you an actual, current, reproducible answer for your specific interpreter version and workload shape, which is the only way to know for certain, since relative performance between approaches can and does shift across Python versions as the interpreter itself evolves (recall the JIT and adaptive interpreter improvements from the What's New in Python 3.13+ lesson).

The discipline this section builds toward, tying profiling, memory profiling, and benchmarking together: performance claims in professional Python work should be backed by a specific, reproducible measurement — 'I profiled this and cumulative time showed X', 'I benchmarked both implementations with timeit and A was Y% faster on this workload' — rather than general folklore about what's supposedly fast or slow in Python, which is frequently outdated, workload-dependent, or simply wrong for your specific case.

Wrapping code in time.time() before and after feels like benchmarking. It usually isn't — a single run is dominated by noise you haven't accounted for.

A single time.time() measurement is noisy -- OS scheduling, caching, and background processes all add variance a single run can't average out.

Checkpoint: Why does a single time.time() measurement typically give a different number on each run of the exact same code?

• →OS scheduling, caching effects, and background system activity introduce variance a single measurement cannot average out
• →Python's interpreter itself runs code at a randomly varying speed by design

timeit.timeit() runs code MANY times automatically and reports something far more stable and comparable than a single measurement.

Checkpoint: What does running code 1000 times and averaging (as timeit does) accomplish that a single measurement cannot?

• →It averages out random measurement noise, producing a far more stable, representative estimate of the code's true typical speed
• →The code itself runs measurably faster the more times it is repeated

To compare two implementations fairly, benchmark both under IDENTICAL conditions -- same input, same machine state, same run count.

Benchmarking gives you rigorous measurement; Performance Optimization closes this section with the concrete techniques you apply once profiling and benchmarking have told you where to focus.

Average Real Benchmark Runs. Finish average_of_runs(): averaging several runs smooths out single-measurement noise.