HTTP pipeline guide
Convert curl headers to JSON without mixing in the body
Choose the curl capture mode before choosing the parser. --head sends an HTTP HEAD request; --verbose traces an otherwise normal request to stderr; --dump-header records received response headers without changing the method. jc-rs turns the resulting message blocks into typed JSON while keeping redirects and recognized repeated headers visible.
By Oleg Sotnikov · Published and checked · Output shapes verified against the jc-rs curl and HTTP-header fixtures
Three curl options answer different questions
“Show me the headers” is ambiguous. A server may handle HEAD differently from GET, and a verbose trace contains the outgoing request as well as every incoming response. Pick the row that matches what you need to observe.
| Option | Method | Captured data | Use it when |
|---|---|---|---|
| --head (-I) | HEAD | Response headers on stdout | You genuinely want HEAD semantics and no body |
| --dump-header (-D) | Unchanged | Received response headers to a file or stream | You need headers from the real GET, POST, or other request |
| --verbose (-v) | Unchanged | Request and response trace on stderr; body on stdout | You need both sides of the exchange for debugging |
curl's --json option is unrelated: it is shorthand for sending a JSON request body and content headers. It does not convert response headers or the response body to JSON.
Parse a real HEAD response from stdout
For HTTP, --head (or -I) asks curl to issue HEAD. The response has headers but no response body, so stdout is already a clean parser input.
#!/usr/bin/env bash
set -Eeuo pipefail
curl --silent --show-error --head https://example.com/ |
jc-rs --curl-head |
jq '.[0] | {
status: ._response_status,
content_type: .["content-type"],
content_length: .["content-length"]
}'This measures the HEAD endpoint, not “GET without downloading the body.” That distinction matters when an application generates headers dynamically, a CDN handles methods differently, or an origin has incomplete HEAD support.
Use --dump-header when the actual method matters
--dump-header (or -D) writes received protocol headers to a destination you choose. It does not turn GET into HEAD. Keep the body and headers in separate files, then parse the header artifact after curl succeeds.
url=https://example.com/
curl --silent --show-error \
--dump-header response.headers \
--output response.body \
"$url"
jc-rs --http-headers < response.headers > response-headers.jsonA clean header file can use the lower-level HTTP headers parser. The curl head parser accepts it too; its extra job is stripping verbose prefixes and informational lines.
Avoid curl --include as parser input when you also need the body. It places headers and body on the same stdout stream; a body can contain text that looks exactly like another HTTP message.
curl -v is a trace on stderr
Verbose mode does not mean “print response headers to stdout.” curl writes the body to stdout as usual and sends protocol trace lines to stderr. Outgoing lines begin with > , incoming lines with < , and connection notes with * .
curl --verbose https://example.com/ | jc-rs --curl-head#!/usr/bin/env bash
set -Eeuo pipefail
url=https://example.com/
trace_file=$(mktemp)
trap 'rm -f -- "$trace_file"' EXIT
if ! curl --silent --show-error --verbose \
--output /dev/null \
"$url" 2>"$trace_file"; then
cat "$trace_file" >&2
exit 1
fi
jc-rs --curl-head <"$trace_file" |
jq 'map(select(._type == "response"))'jc-rs removes curl's incoming and outgoing prefixes, ignores recognized connection and timing lines, and delegates the remaining messages to the HTTP header parser. A verbose exchange can therefore produce request and response objects in wire order.
Repeated headers need header-specific rules
JSON cannot retain two useful values under one key without choosing an array or another representation. Blind comma splitting is wrong too: separate Set-Cookie lines are distinct values, and a cookie expiry date itself contains a comma.
printf '%s\r\n' \
'HTTP/1.1 302 Found' \
'Location: https://example.com/final' \
'Set-Cookie: theme=dark; Path=/' \
'Set-Cookie: session=abc123; Path=/; HttpOnly' \
'Cache-Control: no-store' \
'Cache-Control: private' \
'' |
jc-rs --curl-head |
jq '.[0] | {
status: ._response_status,
location,
cookies: .["set-cookie"],
cache_control: .["cache-control"]
}'{
"status": 302,
"location": "https://example.com/final",
"cookies": [
"theme=dark; Path=/",
"session=abc123; Path=/; HttpOnly"
],
"cache_control": ["no-store", "private"]
}The current schema accumulates set-cookie, cookie, and content-security-policy fields without comma splitting. It aggregates and splits a maintained set of list-valued headers such as cache-control, vary, and www-authenticate.
Only the maintained header sets get this treatment. For an ordinary header outside those sets, a later occurrence currently replaces the earlier one. If an extension header may repeat and every instance matters, retain the raw artifact and add a tested rule before treating the JSON as lossless.
A redirect chain is several responses
Without --location, curl stops at the first redirect. With it, header capture contains a response block for every hop. Keep those blocks separate: merging would attach an early response's cookies or cache policy to the final resource.
#!/usr/bin/env bash
set -Eeuo pipefail
curl --silent --show-error --location \
--dump-header - \
--output /dev/null \
http://example.com/ |
jc-rs --curl-head |
jq 'map(
select(._type == "response")
| {
status: ._response_status,
location: (.location // null),
content_type: (.["content-type"] // null)
}
)'Combining --head --location produces a chain of HEAD requests; use it only when HEAD behavior is the question. Proxies, authentication handshakes, and informational 1xx responses can also add message boundaries, so select the final response instead of assuming index zero.
jq '{
chain: [ .[] | select(._type == "response") ],
final: ([ .[] | select(._type == "response") ] | last)
}' response-exchange.jsonUse the exact jc-rs header field names
Each HTTP message becomes one object. Header names are lowercase and remain kebab-cased. Use bracket syntax in jq for a hyphenated name: .["content-type"] is a field lookup, while a bare hyphen can be parsed as subtraction.
_typerequest or response
Distinguishes message kinds in a verbose trace
_request_method / _request_uristring
Present on request objects
_response_statusnumber
HTTP status code on response objects
content-length, agenumber when parseable
Selected numeric headers are typed
set-cookiearray
Each cookie header stays separate
date_epoch_utcnumber
Added beside a recognized HTTP date
Inspect the fixture-backed example on the HTTP headers parser page before writing a downstream jq contract.
Separate HTTP status, process failure, and secrets
By default, an HTTP 404 can still be a successful curl transfer. The parsed _response_status reports the HTTP result; curl's process status reports whether the transfer completed under its options. Check both when the job cares about both.
set -Eeuo pipefail
curl --silent --show-error --location \
--dump-header - \
--output /dev/null \
https://example.com/ |
jc-rs --curl-head |
jq -e '
[ .[] | select(._type == "response") ]
| last
| select(._response_status >= 200 and ._response_status < 300)
' > final-response.jsonVerbose traces and parsed JSON can contain Authorization, Cookie, Set-Cookie, signed URLs, and internal hostnames. Protect raw captures and delete sensitive fields before sending JSON to logs, tickets, or analytics.
jc-rs --curl-head < curl.trace |
jq 'map(del(
.authorization,
.cookie,
.["set-cookie"],
.["proxy-authorization"]
))' > safe-for-review.jsoncurl manuals and parser source
Capture semantics come from the official curl command-line manual and curl HTTP scripting guide. Prefix removal, boundaries, conversions, and repeated-header behavior are checked against the curl parser and HTTP header parser source.