NDJSON vs JSON vs JSONL

JSON or NDJSON: a working decision table

Choose based on how the data moves. The same record schema can travel in either container.

The differences that matter in production

Latency and backpressure

An NDJSON producer can hand off a record as soon as that record is complete. The downstream process can then slow the producer through the pipe or socket instead of accepting an entire collection first. Framing alone does not guarantee low latency, though. User-space and pipe buffers still exist, so live producers need an explicit flush policy.

Memory

NDJSON makes bounded-memory code natural: read one line, parse one value, do the work, discard it. A specialized streaming JSON parser can also walk a large array incrementally, so JSON does not inherently require loading the whole file. Many everyday APIs and libraries do load a JSON document in one call, which is where the practical difference appears.

Failure recovery

A malformed NDJSON line has an obvious boundary. A reader may stop, quarantine that line, or report it and continue. The format does not choose the policy for you. With one JSON document, a syntax error can make the whole document invalid, even when most records look intact.

Appending, concatenating, and splitting

Appending a complete line to a newline-terminated NDJSON file preserves its structure. Concatenating two correctly newline-terminated NDJSON files does too. Two JSON arrays cannot be concatenated into a valid single document, and appending an item means editing the surrounding array. Line-based tools can also split NDJSON at record boundaries, provided records never contain physical newlines.

Schema evolution

Neither format supplies a schema. If fields change over time, put a version in each record or version the stream contract outside the file. Per-record framing makes mixed versions possible, but it does not make them safe automatically.

Produce NDJSON from command output with jc-rs

jc-rs streaming parsers have names ending in -s. They consume input line by line and, by default, write one compact JSON record per line. The -u option means “unbuffer”: it flushes stdout after every emitted record, which matters for a pipe that stays open.

tail -f /var/log/nginx/access.log \
  | jc-rs -u --clf-s \
  | jq -c 'select(.status >= 500)'

Each part has a separate job. --clf-s recognizes Common or Combined Log Format, -u makes records visible immediately, and jq -c filters while keeping every result on one line. Leaving off -u does not change the NDJSON shape; it only allows buffered writes.

A finite file does not need per-record flushing

cat access.log | jc-rs --clf-s > access.ndjson

The output remains one JSON object per line. Letting jc-rs buffer writes avoids a flush for every record and is the sensible default when the input will close.

Keep going after a bad streaming record

ls -l | jc-rs -qq --ls-s > checked.ndjson
jq -c 'select(._jc_meta.success == false)' checked.ndjson

By default, a streaming parse error stops the run. With -qq, jc-rs continues and adds _jc_meta.success to emitted records. A failed line becomes its own error record with the original line and error text, so the gap is visible rather than silently dropped. This is parser-specific: parsers that deliberately preserve unknown input in an unparsable field still consider that record successfully handled. Check the streaming parser's schema before routing failures.

Convert JSON to JSONL, and JSONL back to JSON

If the source JSON is a top-level array, jq can write each element as one compact line:

jq -c '.[]' records.json > records.jsonl

The reverse operation slurps every JSON value from the input stream into an array:

jq -s '.' records.jsonl > records.json

jq -s holds the collected values in memory. That is fine for a bounded file that fits comfortably, but it removes the memory advantage of record-at-a-time processing. If the next system accepts NDJSON, keep the stream line-delimited instead of building an array only to split it again.

Four easy ways to break a JSON Lines pipeline

1. Pretty-printing the stream. Multi-line indentation destroys the physical line boundary. Keep the transport compact and pretty-print only the record you are inspecting.
2. Writing logs to stdout beside JSON. A progress message becomes a malformed record. Send diagnostics to stderr and reserve stdout for data.
3. Forgetting the final newline. Many readers accept the last line without one, but a trailing newline makes safe concatenation and shell processing less surprising.
4. Assuming framing is validation. One line can still contain invalid JSON or a valid value with the wrong fields. Validate syntax and schema at the boundary that owns the contract.

Common questions