Every format covered in this section ā CSV, JSON, XML, ZIP, binary ā eventually needs to handle a file too large to load entirely into memory. This lesson closes the section with the general, format-agnostic streaming patterns that make file size irrelevant to memory usage.
1The Core Distinction: Eager Loading vs Lazy Iteration
f.read() is an explicit request: 'give me the entire file's contents, right now, as one complete string (or bytes object).' For a file whose size is a meaningful fraction of, or exceeds, available system memory, this single call can exhaust memory before your program even begins processing anything ā the eager, all-at-once nature of .read() is precisely the problem, independent of how efficiently the *rest* of your code processes the data afterward.
for line in f:, by contrast, uses the file object's built-in status as a lazy iterator (exactly the iterator protocol covered in depth in the Iterators lesson) ā each iteration of the loop pulls exactly one line from the underlying file, processes it, and only then requests the next one. At no point does the entire file's content exist simultaneously in memory; the memory footprint is bounded by the size of a single line (plus whatever your processing logic itself retains), regardless of whether the total file is ten lines or ten billion.
This is the same core principle established in the Generators lesson, applied specifically to files: a file object *is*, itself, a generator-like lazy iterator by default, and treating it as one ā rather than immediately materializing its full contents with .read() ā is what makes file size a non-issue rather than a scaling concern that needs to be specially handled once files get 'too big.'
# Loads the WHOLE file into memory before processing anything
with open("huge_log.txt") as f:
content = f.read() # could be gigabytes, all at once
lines = content.split("\n")
error_lines = [l for l in lines if "ERROR" in l]One line in memory at a time ā file size is irrelevant to memory usage
2Chunked Reading for Binary Data With No Natural Line Breaks
Line-by-line iteration works naturally for text files because lines are a genuine, meaningful unit of the data ā but binary files (images, compiled data, arbitrary binary formats from the previous lesson) have no equivalent natural boundary; there's no notion of a 'line' to iterate over. The equivalent streaming pattern for binary data is chunked reading: f.read(chunk_size) requests a specific, bounded number of bytes rather than the entire file, and calling it repeatedly in a loop processes the file incrementally, one fixed-size chunk at a time.
while chunk := f.read(chunk_size): uses the walrus operator (:=, from the Structural Pattern Matching module's broader coverage of modern Python syntax, though introduced earlier in 3.8) to assign the result of f.read(chunk_size) to chunk *and* use that same value as the loop's truthiness condition in one expression ā f.read() returns an empty bytes object (falsy) once the end of the file is reached, naturally terminating the loop at exactly the right point without a separate explicit check.
The chunk size itself is a tunable parameter balancing two competing concerns: too small (a few bytes at a time) incurs meaningful per-call overhead from the sheer number of read operations; too large partially reintroduces the memory problem chunking exists to avoid. A chunk size in the range of tens of kilobytes to a few megabytes is a common, reasonable default for most binary streaming use cases, though the specific right value depends on the data and can be determined empirically using the benchmarking techniques from the Python Performance section.
# Constant memory, regardless of the file's total size
with open("huge_log.txt") as f:
error_lines = [line for line in f if "ERROR" in line]
# f itself is an iterator -- 'for line in f' pulls one line at a timeProcesses any file size with a fixed, bounded memory footprint
3Composable Streaming Pipelines: Generators All the Way Down
The most powerful expression of this section's streaming discipline is chaining multiple generator functions together into a pipeline, exactly mirroring the itertools composability covered earlier in this curriculum. parse_lines(path) is a generator yielding each stripped line from a file; filter_errors(lines) is a *separate* generator that takes any iterable (including another generator) and yields only the lines matching a condition. Composing them ā filter_errors(parse_lines("huge_log.txt")) ā creates a pipeline where data flows through both stages lazily, one line at a time, with no intermediate list ever materialized at any stage.
This composability is what makes generator-based pipelines genuinely more powerful than a single large function trying to do everything at once: each stage (parse_lines, filter_errors, and potentially further stages ā a transform_lines, an aggregate_stats) is independently testable, independently reusable, and can be recombined in different orders or combinations for different processing needs, all while preserving the exact same constant-memory guarantee regardless of how many stages are chained together.
This pattern ā read lazily, process lazily, compose lazily ā is the complete, general answer to 'how do I handle a file too large to fit in memory,' and it applies uniformly across every format this section has covered: a streaming CSV pipeline, a JSON Lines pipeline, a chunked binary pipeline, all built from the identical underlying discipline of never materializing more of the data than the current processing step genuinely needs at that instant.
def process_large_binary(path: str, chunk_size: int = 1024 * 1024):
with open(path, "rb") as f:
while chunk := f.read(chunk_size): # walrus operator: assign AND check truthiness
process_chunk(chunk) # never holds more than one chunk in memoryFully lazy, end to end ā no intermediate list, ever
4Step-by-Step Breakdown
A file twice the size of your available RAM is not a special case requiring special code ā if you stream it correctly from the start, size becomes irrelevant. Let's build that discipline.
f.read() loads the ENTIRE file into memory at once -- fine for small files, a genuine problem for anything approaching or exceeding available RAM.
Iterating a file object directly reads it lazily, one line at a time -- exactly the generator pattern from the Generators lesson, applied to files.
Checkpoint: Why does "for line in f" use constant memory regardless of the file's total size, while f.read() does not?
- āA file object is itself a lazy iterator that yields one line at a time; f.read() eagerly loads the entire file at once
- āIterating a file automatically compresses its contents in memory
For binary data with no natural line breaks, read fixed-size CHUNKS in a loop instead -- the same constant-memory principle, applied to binary.
Combine streaming with a generator PIPELINE -- filter, transform, and aggregate a huge file in one lazy pass, exactly like the itertools composition from earlier.
Checkpoint: In the parse_lines/filter_errors pipeline, does filter_errors wait for parse_lines to finish producing ALL lines before it starts filtering?
- āNo ā both are generators, so filter_errors processes each line as parse_lines yields it, one at a time, fully lazily
- āYes ā parse_lines must complete fully before filter_errors can begin
That completes File Processing ā CSV, JSON, YAML, XML, ZIP, binary formats, and now the streaming discipline that makes file size a non-issue across all of them. Next, Networking covers talking to the outside world over HTTP and WebSockets.
Stream Real Data in Chunks. Finish read_in_chunks(): fixed-size chunks keep memory usage constant regardless of total size.
Level Up š
Advanced cheat sheets, SEO tricks, and interview prep for this topic.
Browser Support
Fully supported (via server-side Python execution).
Fully supported (via server-side Python execution).
Fully supported (via server-side Python execution).
Fully supported (via server-side Python execution).
Best Practices
Default to iterating a file object directly (for line in f) instead of f.read() for any file of meaningful or unknown size
This costs nothing when the file happens to be small, and scales correctly with zero code changes if the file later turns out to be far larger than expected.
Build multi-stage processing as composed generator functions rather than one function that materializes intermediate lists
Chaining generators (parse ā filter ā transform) preserves constant memory usage end to end and keeps each stage independently testable and reusable, exactly like itertools composability.
Frequent Bugs
Calling f.read() (or list(f)) on a file whose size isn't known in advance or could grow over time, assuming it will always be small enough to fit in memory, and discovering the assumption breaks once real production data grows.
Default to lazy iteration (for line in f, or chunked reads for binary data) for any file whose size isn't guaranteed to stay small permanently ā it costs nothing extra for small files and scales correctly as data grows.
Real-World Examples
A Composable Log-Processing Pipeline for Multi-Gigabyte Files
A monitoring tool needs to parse, filter, and summarize error patterns from log files that can grow to tens of gigabytes, on a server with limited available memory.
def read_lines(path):
with open(path) as f:
for line in f:
yield line.rstrip("\n")
def parse_log_entries(lines):
for line in lines:
if entry := parse_log_line(line):
yield entry
def filter_errors(entries):
for entry in entries:
if entry.level == "ERROR":
yield entry
for error in filter_errors(parse_log_entries(read_lines("huge.log"))):
report(error) # entire pipeline: constant memory, any file size