Skip to content
jc-rs
GitHub

Shell pipeline guide

Use jc-rs and jq safely in Bash

jq reads JSON; it does not parse the columns printed by ps, df, or ss. When the command has no suitable JSON mode, jc-rs establishes the schema before jq runs. A production Bash script also needs to preserve quoting, keep stderr out of the data, and notice failures from every stage.

By · Published · Commands checked with Bash, jq 1.7, and the jc-rs CLI

Producercommand stdout
Parserjc-rs
Queryjq
ConsumerJSON / text

jc-rs creates JSON; jq queries it

A reliable Bash and jq pipeline has three contracts. The command writes the format its parser expects. jc-rs converts that format to a documented JSON schema. jq selects, reshapes, or renders values from that schema.

Bash
LC_ALL=C ps aux |
  jc-rs --ps |
  jq 'map(select(.mem_percent >= 5))'
ps aux

Process collection

Not: JSON quoting or field names

jc-rs --ps

Columns, types, nulls, and JSON serialization

Not: Which processes you want

jq

Filtering and output shape

Not: The human-readable ps grammar

If the producer has a stable native JSON flag, prefer it and omit jc-rs. The extra parser is useful only when human-readable output is the available interface.

Decide whether the next stage needs an array

Most non-streaming command parsers return an array of records. jq can preserve that array for another JSON consumer, or unwrap it into a stream for text processing.

map(select(.use_percent >= 80))

one JSON array

API payload, file, or another JSON stage

.[] | select(.use_percent >= 80)

successive JSON objects

streaming into another jq-aware command

-r '.[] | .mounted_on'

raw text lines

human output or a line-oriented shell command

-c '.[]'

compact JSON, one value per line

NDJSON output

Keep the matching rows as JSON
LC_ALL=C df -h |
  jc-rs --df |
  jq 'map(select(.use_percent >= 80))'
Render two fields as tab-separated text
LC_ALL=C df -h |
  jc-rs --df |
  jq -r '.[]
    | select(.use_percent >= 80)
    | [.mounted_on, ((.use_percent | tostring) + "%")]
    | @tsv'

Use -r only at the boundary where JSON becomes text. Keeping JSON longer avoids a second round of shell splitting and escaping.

Single-quote the jq program; pass data with arguments

In Bash, a jq program normally belongs in single quotes. That prevents the shell from expanding $variables, backslashes, and wildcard characters before jq sees them. Values from Bash should cross the boundary with --arg or --argjson, never by building jq source code.

String input

Bash
wanted_user=alice

LC_ALL=C ps aux |
  jc-rs --ps |
  jq --arg user "$wanted_user" 'map(select(.user == $user))'

Numeric input

Bash
limit=${LIMIT:-80}
case $limit in
  (''|*[!0-9]*) printf 'LIMIT must be an integer\n' >&2; exit 2 ;;
esac

LC_ALL=C df -h |
  jc-rs --df |
  jq --argjson limit "$limit" 'map(select(.use_percent >= $limit))'

Avoid

Building a double-quoted jq program that interpolates $wanted_user.

Quotes or jq syntax inside the value can break the program or change its meaning.--arg serializes it as data instead.

Make an upstream failure fail the script

Bash normally reports the status of the last command in a pipeline. If the producer or jc-rs fails but jq exits successfully, the script can appear healthy. Enable pipefail and use jq's -e mode when the jq result itself is a condition.

Fail if any stage fails
#!/usr/bin/env bash
set -Eeuo pipefail

if ! {
  LC_ALL=C df -h |
    jc-rs --df |
    jq -e '
      if type == "array"
      then map(select(.use_percent >= 90))
      else error("expected an array from the df parser")
      end
    '
} >full-filesystems.json 2>pipeline.err; then
  printf 'disk report failed; see pipeline.err\n' >&2
  exit 1
fi

set -e alone is not a substitute for pipefail. An empty JSON array is also a valid, truthy jq value; test its length separately if “no records” is an error for your job.

Keep stderr separate from structured data

Pipes carry stdout. That is exactly what you want: diagnostics remain on stderr while parseable data moves right. Redirect the whole pipeline's stderr to a separate file when a scheduled job needs a record of failures.

Good

{ command | jc-rs --parser | jq ...; } 2>pipeline.err

The parser receives only the command's stdout.

Bad

command 2>&1 | jc-rs --parser

An error message is mixed into the data and may become a bogus row.

Locale is part of the input contract too. For commands whose headings or numbers are localized, set LC_ALL=C on the producer, as the examples above do. It does not need to be exported for the whole script.

Define what empty input means for this job

Parsers do not all treat empty input identically: for some formats an empty collection is meaningful; for others it is malformed input. After a successful parse, decide whether zero records is acceptable rather than relying on implicit jq behavior.

Require at least one matching process
if ! LC_ALL=C ps aux |
  jc-rs --ps |
  jq -e --arg user "$wanted_user" '
    map(select(.user == $user))
    | if length > 0 then . else error("no matching process") end
  ' >processes.json
then
  printf 'process lookup failed or returned no rows\n' >&2
  exit 1
fi

For one scalar destined for a shell variable, use raw output and require a value. Keep the expansion quoted afterward:

Capture one PID
pid=$(LC_ALL=C ps aux |
  jc-rs --ps |
  jq -er --arg user "$wanted_user" '
    first(.[] | select(.user == $user)) | .pid
  ')

printf 'first PID: %s\n' "$pid"

The same rules apply to a live stream

Streaming parsers produce NDJSON: each record is a complete JSON value. jq naturally reads that sequence. Use compact output to preserve the one-record-per-line contract.

Bash
tail -F /var/log/nginx/access.log |
  jc-rs -u --clf-s |
  jq -c 'select(.status >= 500)'

Because this pipeline is intentionally long-lived, its final status is available only when it exits. Run production monitors under a supervisor that restarts failed processes and captures stderr.

Check the parser schema before writing jq

The parser pages show fixture-derived field names and output shapes. Check one before committing a jq expression to a script, especially when the source command varies by operating system.

Install jc-rs