multiprocessing gets around the GIL the most direct way possible: instead of multiple threads sharing one interpreter, it runs multiple separate Python interpreter processes, each with its own GIL, genuinely running in parallel on separate cores. This lesson covers the Pool API and the real trade-offs ā inter-process communication is not free.
1Separate Processes, Separate GILs, Real Parallelism
The previous lesson established that threading's fundamental limitation for CPU-bound work is the GIL ā one interpreter, one lock, one thread executing bytecode at a time. multiprocessing sidesteps this limitation at its root by not sharing an interpreter at all: Pool(processes=4) launches four entirely separate operating-system processes, each running its own independent Python interpreter with its own independent GIL.
Because these are genuinely separate processes ā not threads within one process ā the operating system schedules them across separate CPU cores exactly like it would any other unrelated programs running simultaneously, with no GIL contention between them whatsoever. This is why cpu_heavy finally gets a real speedup under multiprocessing where it got none under threading: four processes on four cores are four times the actual computational throughput, not four threads competing for one core's worth of GIL-gated execution time.
Pool.map() is the most common entry point: it distributes an iterable's items across the pool's worker processes, collects each worker's return value, and returns them in the original order ā conceptually similar to a built-in map(), but with the work genuinely distributed across multiple cores instead of executed sequentially.
from multiprocessing import Pool
def cpu_heavy(n: int) -> int:
return sum(i * i for i in range(n))
if __name__ == "__main__":
with Pool(processes=4) as pool:
results = pool.map(cpu_heavy, [20_000_000] * 4)
# Genuinely runs on 4 CPU cores simultaneously -- real speedup this time4 separate interpreters, 4 separate GILs, real multi-core speedup
2The Real Cost: Processes Do Not Share Memory
The parallelism multiprocessing provides is not free ā it's traded directly against the very thing that made threads lightweight: shared memory. Threads within one process share the same address space, so passing a large list to a thread costs nothing beyond passing a reference. Separate processes have entirely separate memory spaces by default, so any data crossing a process boundary ā arguments passed to a worker, the return value coming back ā must be pickled (Python's built-in serialization format), transmitted through an OS-level pipe or socket, and unpickled back into a usable object on the receiving end.
This serialization cost is proportional to the size of the data being transferred, and for genuinely large datasets it can dominate ā or even exceed ā the time saved by parallelizing the actual computation. A function that processes a 10-million-element list gains little from multiprocessing if pickling and transmitting that list back and forth takes longer than the parallel speedup saves; the calculus only favors multiprocessing when the computation per unit of data is expensive relative to the data's own size.
For cases where genuine shared memory is needed despite using separate processes, multiprocessing.shared_memory and multiprocessing.Value/Array provide narrow, specific mechanisms for sharing data without full pickle-based serialization ā but they're a deliberate, more complex opt-in, not the default behavior, precisely because unrestricted shared memory across processes reintroduces the exact synchronization hazards (race conditions) that separate memory spaces avoid by construction.
import time
from multiprocessing import Pool
def process_large_list(data: list) -> int:
return sum(data)
# Each call: the ENTIRE `data` list is pickled, sent to the worker
# process, and the result is pickled back -- overhead grows with size
with Pool(4) as pool:
result = pool.apply(process_large_list, ([1] * 10_000_000,))Pickled, piped, and unpickled ā real overhead scaling with size
3The if __name__ == "__main__": Guard Is Not Optional
On Windows and on macOS (using the default 'spawn' start method), creating a new process works by launching a fresh Python interpreter and having it re-import the main module from scratch ā there's no fork()-based memory copy to rely on, since spawn creates a genuinely new process from zero. If module-level code that creates a Pool and starts work isn't guarded behind if __name__ == "__main__":, that guard-less code re-executes every single time a new worker process imports the module ā including, recursively, Pool(4) creating four more processes inside each of the four processes it just created, and so on, exhausting system resources almost immediately.
The if __name__ == "__main__": guard works because __name__ is only set to "__main__" in the process that was originally invoked directly (e.g. python script.py); when a worker process re-imports that same file as a module (to access the function being parallelized), __name__ is the module's actual name, not "__main__", so the guarded block simply doesn't re-execute. This is precisely why worker functions passed to Pool.map() must be defined at module level (importable by name) rather than as local closures ā the worker process needs to be able to import and locate that exact function during its own separate startup.
On Linux, the default 'fork' start method copies the parent process's memory directly rather than re-importing, which historically made the guard *appear* optional there ā but relying on that platform difference produces code that silently breaks the moment it runs on Windows or macOS, so the guard should be treated as required on every platform, not just the ones where skipping it happens to fail loudly.
# WITHOUT the guard, on spawn-based platforms, each new process
# re-imports this module, which would re-run Pool(4) again, recursively:
from multiprocessing import Pool
def work(x): return x * 2
if __name__ == "__main__": # REQUIRED
with Pool(4) as pool:
print(pool.map(work, range(10)))Required on spawn-based platforms ā prevents recursive process creation
4Step-by-Step Breakdown
Threading couldn't speed up our CPU-bound loop. multiprocessing can ā because it sidesteps the GIL by not sharing an interpreter at all.
multiprocessing.Pool distributes work across multiple separate Python PROCESSES, each with its own interpreter and its own GIL.
Checkpoint: Why can multiprocessing achieve genuine parallelism for CPU-bound work when threading cannot?
- āEach process runs its own separate Python interpreter with its own GIL, so there is no single lock being contended
- āmultiprocessing makes the CPU itself run instructions faster
Data passed between processes must be PICKLED (serialized) and sent over an OS-level pipe ā this has a real cost, unlike sharing memory directly between threads.
Checkpoint: Why does passing a very large list to a worker process have real overhead that passing it to a thread would not?
- āThe data must be pickled (serialized) and sent through an OS pipe, since processes don't share memory
- āmultiprocessing automatically compresses data, which adds CPU time
The if __name__ == "__main__": guard isn't optional on Windows/macOS spawn-based process creation -- it prevents infinite recursive process creation.
Threads for I/O, processes for CPU ā asyncio is next, a third model built specifically for I/O-bound work at much higher concurrency than threading.
Distribute Real CPU-Heavy Work. Finish parallel_map_simulation(): Pool.map applies a function to every input.
Level Up š
Advanced cheat sheets, SEO tricks, and interview prep for this topic.
Browser Support
Fully supported (via server-side Python execution).
Fully supported (via server-side Python execution).
Fully supported (via server-side Python execution).
Fully supported (via server-side Python execution).
Best Practices
Reach for multiprocessing specifically for CPU-bound work where per-item computation dominates data transfer cost
The serialization overhead of passing data between processes means multiprocessing pays off most clearly when each unit of work involves substantial computation relative to the size of the data being processed.
Always guard process-creating code with if __name__ == "__main__":, on every platform
Relying on Linux's fork-based behavior to skip the guard produces code that silently breaks (or explodes into recursive process creation) the moment it runs on Windows or macOS's spawn-based default.
Frequent Bugs
Parallelizing a function that processes very large in-memory data structures with multiprocessing, then being surprised the parallel version is slower than the sequential one due to pickling overhead.
Profile the actual serialization cost versus computation time before assuming multiprocessing will help; for very large shared data, consider multiprocessing.shared_memory or restructuring so each worker reads its own slice from disk/a database instead of receiving it via pickled arguments.
Real-World Examples
Parallelizing Image Processing Across CPU Cores
A batch job needs to apply an expensive computer-vision transform to 10,000 images, where each image's processing is CPU-intensive and images can be processed completely independently.
from multiprocessing import Pool
from pathlib import Path
def process_image(path: str) -> str:
# Expensive, CPU-bound transform
result_path = apply_transform(path)
return result_path
if __name__ == "__main__":
image_paths = [str(p) for p in Path("images").glob("*.jpg")]
with Pool() as pool: # defaults to os.cpu_count() workers
results = pool.map(process_image, image_paths)