JSON file exports in Python: whole-string encoding versus chunks

Problem. Writing a JSON export can create a full encoded string before the file receives any bytes.
Baseline. Encode all 12,000 records, then write that string.
Solution. Write successive chunks from the same JSONEncoder to a file.^[1]
Measured result. Here, the whole-string path took 24.00 ms median and peaked at 1.20 MiB of traced allocations; the chunk path took 180.78 ms and peaked at 0.20 MiB. These are separate timing and tracing runs.

Reproduce the result

"""Compare two ways to write the same JSON document to a regular file."""
import hashlib
import json
import platform
import statistics
import tempfile
import time
import tracemalloc
from pathlib import Path
N = 12_000
REPEATS = 5
records = [{"id": i, "name": f"item-{i:05d}", "active": i % 3 == 0}
           for i in range(N)]
encoder = json.JSONEncoder(ensure_ascii=False, separators=(",", ":"))
def whole(path):
    with path.open("w", encoding="utf-8", newline="\n") as out:
        out.write(encoder.encode(records))
def streamed(path):
    with path.open("w", encoding="utf-8", newline="\n") as out:
        for chunk in encoder.iterencode(records):
            out.write(chunk)
def measure(fn, path):
    samples = []
    for _ in range(REPEATS):
        start = time.perf_counter_ns()
        fn(path)
        samples.append((time.perf_counter_ns() - start) / 1_000_000)
    tracemalloc.start()
    tracemalloc.reset_peak()
    fn(path)
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return [round(x, 2) for x in samples], round(peak / 1_048_576, 2)
with tempfile.TemporaryDirectory() as folder:
    a, b = Path(folder) / "whole.json", Path(folder) / "stream.json"
    whole(a)
    streamed(b)
    assert a.read_bytes() == b.read_bytes()
    assert json.loads(a.read_text(encoding="utf-8")) == records
    assert json.loads(b.read_text(encoding="utf-8")) == records
    checks = [[], [{}], ["é", "\\", None], records[:1]]
    for case in checks:
        assert encoder.encode(case) == "".join(encoder.iterencode(case))
    print(f"runtime={platform.python_implementation()} {platform.python_version()}")
    print(f"platform={platform.platform()}")
    print(f"rows={N} repeats={REPEATS} warmup=1 byte_equal=True edge_cases=4")
    print(f"bytes={a.stat().st_size} sha256={hashlib.sha256(a.read_bytes()).hexdigest()}")
    for name, fn, path in (("whole", whole, a), ("streamed", streamed, b)):
        times, peak = measure(fn, path)
        print(f"{name} ms={times} median_ms={statistics.median(times):.2f} traced_peak_MiB={peak:.2f}")

Run exactly this command from the directory containing the saved file:

python3 json_stream_bench.py

Captured output

runtime=CPython 3.14.9
platform=Linux-6.12.47+rpt-rpi-v8-aarch64-with-glibc2.41
rows=12000 repeats=5 warmup=1 byte_equal=True edge_cases=4
bytes=560891 sha256=97cf1415f9217c52547741006485a8b2901038f245f8a21b14816cff7ef8a414
whole ms=[22.67, 25.47, 23.61, 25.16, 24.0] median_ms=24.00 traced_peak_MiB=1.20
streamed ms=[190.73, 188.42, 180.78, 174.3, 168.25] median_ms=180.78 traced_peak_MiB=0.20

Environment and method

CPython 3.14.9; Linux 6.12.47+rpt-rpi-v8, aarch64, glibc 2.41; Raspberry Pi container. Standard library only. A deterministic list comprehension makes 12,000 records (no random seed), then each strategy writes a separate temporary file in the same temporary directory. One untimed warm-up precedes five timed writes per strategy. The clock is perf_counter_ns; times are milliseconds and the five raw samples appear above.^[3]

Before timing, the program compares every output byte, parses both files back to the input records and checks empty, singleton, escaped and one-record cases. The SHA-256 printed above identifies the 560,891-byte result. Memory tracing begins after the input and encoder exist; each strategy gets a separate untimed traced write. The recorded peak is Python-traced allocations during that write, rather than whole-process RAM.^[2]

What changed

The standard library exposes JSONEncoder.encode for a complete string and iterencode for successive string chunks.^[1] The second path avoids holding the complete encoded string in this measured implementation. The chunk path took longer here. Encoding strategy and write granularity both change, so these timings cannot isolate the cause of that gap. Both paths still hold the entire input list in memory.

Practical applications

A file export from an already built in-memory result set can use chunked writing when temporary encoder allocations matter more than elapsed time. This experiment measures a regular temporary file; it does not measure a network destination or a different record shape.

A streaming HTTP response can send successive encoder chunks without first constructing the full JSON string. Back-pressure, transport buffering and cancellation are outside this file benchmark; response time cannot be inferred from these samples.

A bounded-memory log archive may write encoded chunks to a file-like sink. The input must also be produced incrementally if overall memory is the constraint: the benchmark constructs every record before either write begins. A single malformed value late in the sequence can leave a partially written file, so an atomic replacement policy needs a separate test.

Limits

This is a single-host, synthetic integer/string/boolean workload with five serial trials and no forced disk flush. The file cache, concurrent container activity and implementation details can change timings. Tracemalloc reports traced Python allocations, not resident memory or kernel page cache.^[2] Whole-string encoding won on elapsed time here; a tiny output or different writer can change the trade-off. For a non-serialisable object, both strategies can raise after work has started, but partial-file behaviour differs.

Sources

1. Python 3.14 json encoder
2. Python 3.14 tracemalloc
3. Python 3.14 time