Formats — read one, write another

The question for this lesson: your data arrives as CSV, but the system downstream wants JSON. Or a partner sends a fixed-width mainframe extract, or an HL7 healthcare message. How does one pipeline deal with so many file formats?

The answer is small: a source or output node’s type: names its format, and everything between them is the same format-agnostic record model. So reading one format and writing another is a one-word change.

What you’ll be able to do

• Change a pipeline’s input or output format by changing a node’s type:.
• Name the formats Clinker reads and writes.
• Avoid two format traps: CSV never infers types, and NDJSON is a json option rather than its own format.

The format lives in type:

A source’s type: is the format it reads; an output’s type: is the format it writes. Here a pipeline reads CSV and writes CSV. Run it, and read what comes out.

✋ Predict first

The source type is csv and the output type is csv, a straight pass-through of two columns. What rows come out?

Reveal the expected result

status,lifetime_value
active,80000
inactive,1200
active,4000

pipeline.yaml // editable

reset ⌖ explain ✓ dry-run ▶ run

pipeline: name: customer_export nodes: - type: source name: customers config: name: customers type: csv # the INPUT format path: ./data/customers.csv options: { has_header: true } schema: - { name: status, type: string } - { name: lifetime_value, type: string } - type: output name: results input: customers config: name: results type: csv # the OUTPUT format — change to json to export JSON path: ./output/customers.csv

Now the format point, hands-on: in the runner above, change the output node’s type: csv to type: json and run again. Same rows, same pipeline, but the result comes out as JSON records instead of CSV lines. Nothing else changed, because the rows flowing between the nodes never depended on the format at either end.

🌱 New here? — format-agnostic records

Once a source reads a file, every row becomes the same internal record (named, typed cells, the schema from lesson 01). Transforms, routes, and aggregates work on that, never on raw bytes, which is exactly why swapping the input or output format doesn’t touch the rest of the pipeline.

The format catalog

Clinker reads and writes all of these; you select one with type::

type:	Format	Typical use
csv	comma/delimited text	the everyday default
json	JSON records	APIs, document stores (NDJSON is a json option)
xml	XML documents	legacy systems, configuration feeds
fixed_width	columns by character position	mainframe extracts (needs declared field widths)
x12, edifact	EDI business documents	invoices, purchase orders between companies
hl7	HL7 messages	healthcare/clinical data

The EDI and HL7 formats are real and stable (with some charset/nesting limits) but specialized, so you’ll reach for them only in those industries. csv and json cover most day-to-day work. (There’s also a swift banking format, but it’s a preview, only message blocks 1–5, so treat it as not-yet-ready.)

Clinker’s real order_fulfillment pipeline shows the everyday case: one CSV source, and two outputs in different formats, a CSV of fulfilled orders and a JSON priority report:

- type: output
  name: fulfilled_orders
  input: route_priority
  config: { name: fulfilled_orders, type: csv,  path: ./output/fulfilled_orders.csv }

- type: output
  name: priority_report
  input: route_priority
  config: { name: priority_report, type: json, path: ./output/priority_report.json }

Preview: not ready

There’s an envelope node on the horizon for framing output: wrapping a stream with a header and/or footer (modes preserve and concat, with header: and footer:) so the file that lands has a top and tail around its records. It’s a preview, though: there’s no shipped example to copy from, and its trailer: isn’t wired up yet. So it’s worth knowing the idea exists, but please don’t lean on it in a real pipeline until it’s finished.

Two format traps

✋ Predict first

A teammate writes a CSV source with no schema, expecting Clinker to guess that an `amount` column is a number. And another writes `type: ndjson` for a newline-delimited JSON output. Which of these works?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

Neither.

• CSV does not infer types. A header names the columns, but you must always declare their types in the schema: (lesson 01). There is no “guess the type”: amount is whatever you declare, and you convert with .to_int() in CXL when you need a number.
• There is no top-level ndjson format. Newline-delimited JSON is a setting on the json format (type: json with an options flag), not its own type:. Writing type: ndjson fails with an unknown-format error.

Create: one source, two output formats

Author the output stage yourself. Because records are format-agnostic once read, one source can feed several outputs in different formats, exactly what the order_fulfillment example does. Write two output nodes that read the same upstream node and emit the same rows as CSV and JSON:

- type: output
  name: as_csv
  input: customers
  config: { name: as_csv, type: csv, path: ./output/customers.csv }
- type: output
  name: as_json
  input: customers
  config: { name: as_json, type: json, path: ./output/customers.json }

Both read input: customers, so both receive every row; only the type: differs. The CSV lands as delimited rows; the JSON lands as an array of records:

[
{"status":"active","lifetime_value":"80000"},
{"status":"inactive","lifetime_value":"1200"},
{"status":"active","lifetime_value":"4000"}
]

Nothing upstream changes; the rows never depended on the format at either end. Verify with clinker run your_pipeline.yaml --dry-run (it resolves both output paths) and --explain.

Retrieval checkpoint

✓ Checkpoint

// quick check

You read CSV today but a partner now needs the same output as JSON. What's the change?

• Change the output node's type: from csv to json, and the rest of the pipeline is untouched
• Rewrite every transform to produce JSON
• Add a converter node between every step

// quick check

From lesson 01: a CSV source has a header row with an `amount` column. Does Clinker infer that amount is a number?

• No, CSV never infers types; you declare each column's type in the schema and convert in CXL
• Yes, Clinker reads the values and picks the type automatically
• Only if the column is named `amount`

Common mistakes

• Expecting CSV type inference. Declare every column’s type in the schema:; the header only names columns.
• Writing type: ndjson. There is no top-level ndjson format; NDJSON is an option on type: json.
• Treating swift as ready. It’s a preview (blocks 1–5 only).

You should be able to:

• Change an output's format by editing its type:
• Name at least four formats Clinker reads/writes
• Explain why swapping a format doesn't change the rest of the pipeline
• State the two traps: CSV declares types (no inference); ndjson is a json option

Verify in the checkout:

clinker run customer_export.yaml --dry-run
clinker run order_fulfillment.yaml --explain

Go deeper (optional)

Because every format funnels into the same record model, a pipeline that transforms data is written once and works for any input/output format you point it at. Next, the capstone: build and validate a complete pipeline end to end.