Parse command output reliably

Start with the producer, not a regular expression

Write down the exact invocation before writing parsing code. Options often select a different grammar: ps -ef and ps aux do not merely show more or fewer rows; they select different columns. A parser tested against one cannot silently claim the other.

First look for a structured or purpose-built interface. Use documented native JSON when it exists. For systemd, for example, systemctl status is explicitly human-facing, while systemctl show exposes normalized properties for programs. Only parse display text when it is the artifact or interface you genuinely have.

The native JSON or jc-rs decision guide applies that rule to ip, lsblk, journalctl, and text-only fallbacks.

Locale is part of the bytes you parse

Locale can translate headings and month names, change collation, and choose decimal or thousands separators. A parser that recognizes Filesystem cannot infer that an unfamiliar translated word means the same field. Set the locale on the producer so it emits the grammar represented by your fixtures.

LC_ALL=C df -P |
  jc-rs --df |
  jq 'map({filesystem, mounted_on, use_percent})'

LC_ALL=C has higher precedence than the other locale categories for that process. Applying it to the left side of the pipe is deliberate: df is the program rendering localized text. You do not need to export a process-wide locale for unrelated commands.

Decide whether whitespace is syntax or presentation

Aligned tables use spaces for at least two jobs: separating columns and padding short values. A free-text final column may contain the same spaces. Empty cells, tabs, and a long identifier can shift everything to the right. A blanket split_whitespace() therefore has no way to reconstruct the intended row unless the command's grammar supplies more information.

awk or sed is a sound choice when the producer promises an unambiguous delimiter and constrains the field contents. A short parser is then easier to audit than a large one. Git ref names cannot contain a tab, so this invocation creates a real tab-delimited contract before jq turns each line into an object:

git for-each-ref \
  --format='%(refname)%09%(objectname)' \
  refs/heads/ |
  jq -Rn '[
    inputs
    | split("\t")
    | {ref: .[0], object: .[1]}
  ]'

By contrast, the command column in ps and the description column in systemctl are not safely recovered by splitting every run of spaces. A matching parser uses the known header and row rules for that command. Inspect the fixture-backed schemas on the ps parser and systemctl parser pages before choosing fields downstream.

Make non-interactive output explicit

Many commands inspect whether stdout is a terminal. That decision can enable a pager, color, a tree, shortened headings, ellipses, or width-based wrapping. A cron job and an interactive shell may therefore receive different bytes from the same-looking command.

LC_ALL=C ps auxww |
  jc-rs --ps > processes.json

LC_ALL=C SYSTEMD_COLORS=0 \
  systemctl --all --no-pager --full --plain |
  jc-rs --systemctl > units.json

Use the producer's own options instead of hoping that a large COLUMNS value will suppress every display feature. Then test both piped and pseudo-terminal execution if your application ever runs the command under a terminal allocator.

Treat version drift as a schema migration

Human output changes for good reasons: a new column, clearer units, a renamed state, extra summary lines, or different rendering of missing values. The parser may still return valid JSON while assigning the wrong meaning to one value, which is more dangerous than a clean parse failure.

Record the supported producer versions in test metadata, but assert behavior rather than accepting a version string alone. A distribution may backport output changes without adopting the upstream release number you expected. jc-rs keeps fixtures from several operating systems for precisely this reason.

stderr is evidence, not another input column

A normal pipe carries stdout and leaves diagnostics on stderr. Preserve that split. Redirecting 2>&1 before the parser can turn a permission warning, transient network error, or usage message into a plausible but false record.

Exit status is a separate channel too. Bash normally reports the final command's status for a pipeline, so enable pipefail if an upstream failure must fail the job.

#!/usr/bin/env bash
set -Eeuo pipefail

LC_ALL=C df -P |
  jc-rs --df |
  jq -e 'select(type == "array")' > filesystems.json

curl verbose traces are an intentional exception because curl writes that trace to stderr. Capture it as a named artifact, check curl's status, and only then parse it; the exact pattern is in the curl headers to JSON guide.

Validate the JSON contract, not just JSON syntax

Successful JSON parsing proves that brackets and strings are well formed. It does not prove that the top level is an array, that a utilization field is numeric, that null is allowed, or that the command itself succeeded. Assert the smallest schema the next stage relies on.

#!/usr/bin/env bash
set -Eeuo pipefail

work_dir=$(mktemp -d)
trap 'rm -rf -- "$work_dir"' EXIT

if ! LC_ALL=C df -P >"$work_dir/df.out" 2>"$work_dir/df.err"; then
  cat "$work_dir/df.err" >&2
  exit 1
fi

if ! jc-rs --df   <"$work_dir/df.out"   >"$work_dir/df.json"   2>"$work_dir/jc-rs.err"; then
  cat "$work_dir/jc-rs.err" >&2
  exit 1
fi

jq -e '
  select(
    type == "array" and
    length > 0 and
    all(.[];
      (.filesystem | type == "string") and
      (.mounted_on | type == "string") and
      (.use_percent | type == "number")
    )
  )
' <"$work_dir/df.json" > filesystems.json

This longer form is useful in a scheduled job because it identifies the stage that failed and retains each diagnostic until the script exits. For ordinary interactive work, the shorter pipefail pipeline is often enough. Choose based on the observability the job needs.

Test the assumptions, not just the happy path

One golden sample verifies a happy path. A production parser needs samples selected for the assumptions most likely to be false.

Keep the raw input next to the expected JSON. When a producer changes, the diff then shows whether you are extending a documented grammar or accidentally teaching the parser to accept one machine's corrupted output.