Processing CSV Files in Python

CSV's format, despite looking trivially simple, has a real specification (RFC 4180, loosely followed with variations across tools) specifically because raw data legitimately needs to contain the delimiter character itself — a person's name might contain a comma ("Doe, Jane"), a text field might span multiple lines, and a field might need to contain the quote character itself (escaped as a doubled quote, ""). The convention that handles all of this is wrapping such fields in quotes, and str.split(',') has zero awareness of that convention — it mechanically splits on every comma character, quoted or not, silently producing the wrong number of fields with content shifted into the wrong columns.

This isn't a rare edge case confined to unusual data — addresses, free-text descriptions, and names are all common real-world fields that routinely contain commas, making naive splitting a correctness bug waiting to surface the moment real, messy production data (rather than a clean synthetic test file) is processed. The failure mode is also silent and dangerous specifically because it doesn't crash — it produces plausible-looking, subtly wrong data that can go unnoticed for a long time.

csv.reader(), from the standard library, implements the full quoting-aware parsing logic correctly: quoted fields containing commas, embedded newlines, and escaped quote characters are all handled per the format's actual rules, turning '"Doe, Jane",42,"New York, NY"' correctly into the three intended fields ('Doe, Jane', '42', 'New York, NY') instead of five broken ones.

Opening a CSV file with open("people.csv", newline="") — the empty-string newline argument, not omitting it — is a specific, documented requirement of the csv module, and skipping it is one of the most common CSV-handling mistakes even among experienced developers. By default, open() performs 'universal newline' translation, converting platform-specific line endings (\r\n on Windows) into a uniform \n as the file is read. The csv module needs to see the *raw*, untranslated line endings itself, because it has its own internal logic for correctly handling newline characters that appear legitimately *within* a quoted field (a genuinely multi-line text value) versus newlines that mark the end of a row.

Without newline="", these two newline-handling layers can interfere with each other — on Windows specifically, this commonly manifests as extra blank rows appearing in parsed output, or (more subtly and more dangerously) as incorrect parsing of a field that legitimately contains an embedded newline. This is precisely the kind of detail that works fine in casual testing on one platform and breaks unexpectedly when the same code runs somewhere else, or against data with fields that happen to contain newlines.

The rule to internalize as a fixed habit, not something to re-derive each time: any open() call feeding into or receiving output from the csv module always includes newline="", for both reading and writing, on every platform, unconditionally.

csv.reader() returns each row as a plain list, meaning code has to access fields by positional index (row[0], row[1]) — functional, but fragile: if the CSV's column order ever changes, or if a new column gets inserted, every piece of code relying on positional indexing silently breaks or reads the wrong field, often without any error to signal the mistake. csv.DictReader, by contrast, uses the file's first row as column headers automatically, and returns each subsequent row as a dict keyed by those header names — row["name"], row["age"] — immune to column reordering, and immediately self-documenting about what each field actually represents at the call site.

csv.DictWriter provides the symmetric capability for writing: constructed with an explicit fieldnames list (which also determines the column order in the output file), .writeheader() writes the header row, and .writerow({"name": "Ada", "age": 36}) writes a row from a dict, matching each value to its correct column by key rather than requiring you to remember and maintain the exact positional order the underlying file expects.

This dict-based interface is the professional default for any CSV work involving files with a header row — the positional reader/writer variants remain useful specifically for headerless CSV data, or performance-sensitive code processing an enormous number of rows where avoiding dict construction overhead per row provides a measurable benefit, a case worth confirming via the profiling techniques from the Python Performance section rather than assuming.

line.split(',') looks like it parses CSV. It works right up until a field contains a comma inside quotes — then it silently corrupts your data. Let's use the module built to actually handle this.

line.split(',') breaks the moment a field legitimately CONTAINS a comma inside quotes -- a real, common case in addresses, names, and descriptions.

Checkpoint: Why does line.split(',') produce 5 fields instead of the intended 3 for '"Doe, Jane",42,"New York, NY"'?

• →split(',') has no concept of quoted fields — it splits on EVERY comma, including ones meant to be inside a quoted value
• →This is a known bug in Python's string splitting

csv.reader() correctly handles quoted fields, embedded commas, and embedded newlines -- exactly the cases manual splitting gets wrong.

Checkpoint: Why must open() be called with newline="" when reading or writing CSV files?

• →It prevents Python's universal newline translation from double-processing the line endings the csv module already handles itself
• →It makes reading the file noticeably faster

DictReader/DictWriter use the header row as keys -- code reading by COLUMN NAME instead of fragile positional indexing.

CSV covers tabular data; JSON is the next format every Python developer touches daily, with its own set of correctness details.

Parse Real CSV Rows. Finish parse_csv_rows(): DictReader turns each row into a dict keyed by the header.