Processing JSON Files in Python

JSON's type system is deliberately minimal — strings, numbers, booleans, null, arrays, and objects — which means any Python type outside that set (datetime, Decimal, a custom class instance, a set) has no automatic, unambiguous JSON representation, and json.dumps() raises TypeError the moment it encounters one rather than guessing at a conversion that might be wrong for your specific use case.

The default parameter is the sanctioned extension point for this: a function you provide that json.dumps() calls specifically when it hits an object it can't serialize natively, and whose return value is used in its place (recursively re-serialized if needed). json_default checking isinstance(obj, datetime) and returning obj.isoformat() converts any datetime into a standard, unambiguous, ISO 8601 string representation — a convention widely understood by other tools and languages that will eventually deserialize this same data.

The default function should always end with a raise TypeError(...) for any type it doesn't explicitly handle (rather than silently returning something like str(obj) for everything), so that genuinely unexpected, unhandled types still fail loudly with a clear error rather than being serialized into some plausible-but-wrong representation that might cause quiet data-quality problems downstream.

Any JSON your code parses that didn't originate from your own json.dumps() call — a webhook payload, a file uploaded by a user, a response from a third-party API — should be treated as untrusted input that might not actually be valid JSON at all. json.loads() raises json.JSONDecodeError (a ValueError subclass) when parsing fails, and — following the same precise-exception-catching discipline the Advanced Error Handling section established — catching that specific exception type, rather than a bare except:, ensures you only handle genuine JSON parsing failures without accidentally swallowing unrelated bugs elsewhere in the same function.

JSONDecodeError carries genuinely useful structured attributes beyond its message: .lineno and .colno pinpoint exactly where in the input the parser gave up, and .msg describes what specifically went wrong (an unexpected character, an unterminated string, a trailing comma) — directly analogous to the structured exception attributes covered in the Custom Exceptions lesson, letting calling code (or a log line) report precisely where malformed input failed rather than just 'JSON parsing failed somewhere.'

This matters practically for building resilient systems: a webhook handler receiving occasionally malformed payloads from a third party should log the specific parse failure (with the useful lineno/colno detail) and return an appropriate error response, rather than letting an unhandled JSONDecodeError crash the request handler entirely — exactly the graceful, informative failure handling this platform's error-handling section builds toward generally, applied specifically to JSON parsing.

json.load(f) reads the *entire* file into memory and parses it as one complete operation, returning a single, fully-materialized Python object — perfectly fine for files of a reasonable size, but a genuine problem for a JSON file large enough that loading it entirely could exhaust available memory, since there's no way to get a partial result or process it incrementally with the standard library's json module alone.

For genuinely large JSON files, the common real-world pattern is JSON Lines (also called NDJSON) — a file format where each line is an independent, complete JSON object, rather than one giant JSON array spanning the whole file — which can be processed lazily, one line (one record) at a time, using a generator exactly like the File Processing streaming techniques covered later in this section: for line in f: record = json.loads(line), never holding more than one record in memory at once.

For a file that's genuinely one enormous JSON array or object rather than JSON Lines, and can't be restructured, a streaming JSON parser like the third-party ijson library becomes necessary — it parses incrementally, yielding pieces of the structure as they're encountered rather than requiring the whole document to be read first. Recognizing which situation you're in — 'a reasonably-sized JSON file, json.load() is fine' versus 'a JSON Lines file, iterate lazily' versus 'one enormous JSON document, need a streaming parser' — is the practical judgment call that separates production-ready JSON handling from code that works fine in testing and fails unexpectedly against real production data volumes.

json.dumps() works perfectly until you try to serialize a datetime, and then it just raises TypeError. Let's handle that, and the other real gotchas.

json.dumps() raises TypeError on types it doesn't know how to serialize -- datetime being the most common real-world case.

A custom 'default' function tells json.dumps() how to handle types it doesn't natively understand -- fixing the error cleanly.

Checkpoint: What does passing default=json_default to json.dumps() accomplish?

• →It lets you define how to convert a type json.dumps() does not natively understand into something serializable
• →It makes JSON serialization run measurably faster

json.JSONDecodeError is a SPECIFIC exception -- catch it specifically to handle malformed JSON from external sources gracefully.

Checkpoint: Why catch json.JSONDecodeError specifically, instead of a bare except: around json.loads()?

• →It catches only genuine JSON parsing failures (with useful lineno/colno/msg details) without accidentally hiding unrelated bugs
• →There's no real difference -- both accomplish the same thing

JSON handles hierarchical data cleanly; YAML is the next format, used heavily for human-edited configuration.

Handle Real Malformed JSON. Finish safe_parse_json(): JSONDecodeError lets you handle bad data gracefully.