A generator function looks like a normal function but behaves completely differently: calling it doesn't run the body ā it returns an object that runs the body one step at a time, pausing at each yield. That laziness is what makes generators the right tool for anything too large to hold in memory at once.
1yield Suspends; It Does Not Return
The single most important thing to internalize about yield is that it is not return. return inside a normal function terminates it permanently and hands a value back to the caller; yield hands a value back to the caller too, but *freezes the function's entire execution state* ā the value of every local variable, the current position in the code, the state of any loop ā and waits. Calling next() on the generator object resumes execution from exactly that frozen point, continuing until the next yield or until the function actually returns (implicitly or via a bare return).
This is why calling count_up_to(3) executes nothing: a function containing yield anywhere in its body is compiled by Python as a *generator function*, and calling a generator function never runs its body directly ā it immediately returns a generator object, a lightweight wrapper that will run the body, one next() call at a time, only when asked.
When the function eventually finishes ā the while loop condition becomes false and there's no more yield to reach ā calling next() one more time raises StopIteration, the standard signal used throughout Python's iteration machinery (including for loops, which catch it silently) to mean 'there is nothing left'.
def count_up_to(n: int):
i = 1
while i <= n:
yield i
i += 1
gen = count_up_to(3)
print(gen) # <generator object count_up_to at 0x...> ā nothing has run yet<generator object at 0x...> ā body has not run
2What a for Loop Is Actually Doing
for value in count_up_to(3): is itself syntactic convenience over a manual protocol: Python calls iter() on the generator (a no-op for generators, since they're already their own iterator), then repeatedly calls next() on the result, binding each returned value to value and running the loop body, until next() raises StopIteration ā at which point the loop ends cleanly, with the exception itself caught and silenced by the for statement's own machinery.
This equivalence is worth tracing through manually at least once, because it demystifies not just generators but every iterable object in Python: lists, dicts, files, and range objects are all consumed through this exact same iter()/next()/StopIteration cycle, whether or not their underlying implementation happens to be a generator.
Understanding this also explains a common gotcha: a generator object is exhausted after one full iteration. Once count_up_to(3)'s internal while loop finishes and StopIteration has been raised, calling next() again (or starting a second for loop over the *same* generator object) yields nothing ā you must call count_up_to(3) again to get a fresh generator, since generators, unlike lists, cannot be rewound or reused.
print(next(gen)) # 1 ā runs until first yield
print(next(gen)) # 2 ā resumes after yield, runs to next yield
print(next(gen)) # 3
print(next(gen)) # StopIteration ā the while loop ended1, 2, 3, StopIteration
3Constant Memory: The Actual Reason Generators Matter
The practical payoff of lazy, one-value-at-a-time evaluation is memory. f.readlines() on a 50GB log file attempts to load every single line into a Python list in memory simultaneously ā on most machines, that simply fails or grinds the system to a halt. for line in f: (files are themselves iterators yielding one line at a time) and read_large_file's generator wrapper around it both process the file at a constant, small memory footprint, because only the current line needs to exist in memory at any given moment; the rest of the file stays on disk until requested.
This same principle scales down usefully too: range(10_000_000_000) doesn't allocate ten billion integers ā range is itself a lazy sequence type (not technically a generator, but built on the same 'compute values on demand' philosophy) that computes each integer only when iterated over or indexed.
The professional habit this motivates: whenever a function's job is 'produce a sequence of values that the caller will consume one at a time' ā parsing a file, paginating an API, streaming query results ā default to yield instead of building and returning a full list, unless the caller genuinely needs random access or the full collection's length up front. Generator expressions, (x**2 for x in range(n)), apply the identical laziness to inline expressions without a dedicated function.
for value in count_up_to(3):
print(value)
# 1
# 2
# 3Constant memory (a few KB) instead of loading it all at once
4Step-by-Step Breakdown
range(10_000_000_000) doesn't allocate ten billion integers. Generators are why ā let's see exactly how.
Calling a generator function doesn't execute any of its body ā it returns a generator object, paused before the first line.
Checkpoint: What happens when you call count_up_to(3)?
- āA generator object is returned immediately; no code inside the function has run yet
- āThe entire function body runs immediately, computing all values
Each call to next() resumes execution right where it left off, runs until the next yield, and returns that value.
A for loop calls next() automatically and catches StopIteration for you ā this is what happens under the hood.
The killer feature: memory stays constant no matter how large the sequence is, since only one value exists in memory at a time.
Checkpoint: Why can read_large_file process a 50GB file using only a few KB of memory?
- āOnly one line is held in memory at a time, since yield pauses until the next value is requested
- āPython automatically compresses the file contents in memory
Generators are the most common way to build an iterator ā next, we look at the iterator protocol they implement under the hood.
Yield Real Values Lazily. Finish count_up_to(): yield pauses the function and hands back one value at a time.
Level Up š
Advanced cheat sheets, SEO tricks, and interview prep for this topic.
Browser Support
Fully supported (via server-side Python execution).
Fully supported (via server-side Python execution).
Fully supported (via server-side Python execution).
Fully supported (via server-side Python execution).
Best Practices
Default to a generator for any "produce a sequence" function unless you need random access
yield keeps memory constant and lets consumers stop early (e.g. break out of a for loop) without ever computing the remaining values ā a list comprehension computes everything up front regardless of whether it's all needed.
Remember a generator is single-use; re-call the function for a fresh iteration
Storing a generator object and iterating it twice silently yields nothing the second time. If you need to iterate multiple times, either call the generator function again or materialize it into a list once with list(gen).
Frequent Bugs
Iterating over the same generator object a second time (e.g. passing it to two different functions that each loop over it) and getting silently empty results the second time.
Either call the generator function again to get a fresh generator, or convert it to a list once (results = list(gen)) if you need to iterate the same data multiple times.
Real-World Examples
Paginated API Client as a Generator
A client library needs to expose "all results across every page" as a single iterable, without ever loading every page into memory at once or making the caller manage pagination tokens.
def fetch_all_users(api_client):
page_token = None
while True:
response = api_client.get_users(page_token=page_token)
yield from response["users"]
page_token = response.get("next_page_token")
if not page_token:
break
for user in fetch_all_users(client):
process(user)