Validate before you run

The question for this lesson: you’ve edited a 200-line pipeline. Before you run it over a million rows, how do you check it’s correct, without waiting for the whole run?

What you’ll be able to do

• Use --dry-run to validate config structure and declared types without processing data, and know what it leaves for the real run.
• Use --explain to print the execution plan and read it.
• Decode a diagnostic E-code with explain --code.

Check without running: --dry-run

clinker run customer_etl.yaml --dry-run

--dry-run parses your YAML, checks the config’s structure, validates your declared schema types, and resolves the output paths, then stops, without reading a single data row. If a required field is missing or a schema type doesn’t exist, you find out in a second.

Planning-only --dry-run validates the complete DAG and typechecks CXL against the declared schemas. It catches dangling input: references, unknown fields or methods, and incompatible types before reading records. It cannot prove that the actual file’s bytes satisfy the schema or that your business rule is right.

🌱 New here? — planning versus execution

Planning uses the pipeline and its declarations. To exercise real input, add -n 5: this reads at most five records per Source, evaluates the pipeline, and serializes preview bytes to stdout without opening configured Sink paths. --dry-run-output preview.csv selects an explicit preview file. A sample’s output-row count may differ after filters, aggregates, joins, or fan-out.

Preview limitation at the reference revision

Bounded preview can currently fail with an internal node-buffer cleanup error for the validation and DLQ examples even when their ordinary runs complete. Inspect the exit status and diagnostics. A static expected-result panel is not a successful execution, and a successful sample never proves the entire input.

See the shape: --explain

clinker run customer_etl.yaml --explain

--explain prints the execution plan in labelled sections (the CXL it compiled, the node graph, and how data flows), so you can confirm the pipeline does what you think before you run it. The full output is long; here are the three sections you’ll read most (trimmed):

=== Execution Plan ===

Mode: Streaming
Transforms: 1
Output projections: 1
DAG nodes: 3

=== CXL Expressions ===

Transform 'tagged':
  emit is_active = status == "active"

=== DAG Topology ===

  ● [source] customers
  │ [transform] tagged
  │ [sink] results

✋ Predict first

Read the DAG Topology section. A record enters at the source. Through which nodes does it pass on its way to a file?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

customers (the source) → tagged (the transform) → results (the output). The === DAG Topology === section lists the nodes top-to-bottom in flow order, so you read it straight down. The === CXL Expressions === section above it shows the exact expression Clinker compiled for each transform. Reading both is how you confirm the wiring and the logic match your intent before spending a full run.

Decode a diagnostic: explain --code

When a run or dry-run fails, it prints an E-code. You don’t have to guess what it means; look any code up directly, with no config needed:

clinker explain --code E105

That prints a full explanation of the diagnostic: what it means, an example that triggers it, and how to fix it. E105, for instance, is “Channel binding references undeclared config key”: a channel file bound a config key the target pipeline never declared. The point isn’t to memorize codes. Whenever a run prints one, explain --code turns it into a plain explanation on the spot.

Use: validate a pipeline

Run --dry-run on this one. Predict whether it passes, then check.

✋ Predict first

Read every node name and every input. Does each input match a real node name? Will --dry-run pass or report an error?

Reveal the expected result

=== Resolved Outputs ===

'results' → ./output/customers.csv
    [if_exists=overwrite, write_meta=false]

Dry run: plan valid, 1 inputs, 1 outputs, 1 transforms

pipeline.yaml // editable

reset ⌖ explain ✓ dry-run ▶ run

pipeline: name: customer_etl nodes: - type: source name: customers config: name: customers type: csv path: ./data/customers.csv options: { has_header: true } schema: - { name: status, type: string } - type: transform name: tagged input: customers config: cxl: | emit is_active = status == "active" - type: sink name: results input: tagged config: name: results type: csv path: ./output/customers.csv

Modify: break it on purpose

Now make --dry-run fail in two different ways. In the runner above, try two changes, one at a time, and predict the outcome of each before you re-run.

✋ Predict first

In the pipeline, (a) change a schema field's `type:` to a made-up type like `wizbang`, then dry-run. Then undo that and (b) misspell a transform's `input:` so it names a node that isn't there, and dry-run. For each, does --dry-run catch it, and what exit code do you expect?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

(a) The made-up type: is a declared-type problem, so --dry-run catches it on the spot: the config never compiles and you get exit code 1 (compile error, the job never starts). (b) The misspelled input: is a wiring problem that planning also rejects before execution, with exit 1. Both cases are knowable from the declared pipeline; neither requires data to flow.

// quick check

What does `--dry-run` validate before reading any data?

• Config, schemas, DAG wiring, and CXL field/method/type validity
• Every input row and the correctness of the resulting business data
• Nothing; it just prints the YAML back

Create: pre-flight a pipeline you build

Now put the tools together on a pipeline of your own. Author a small one from scratch: a csv source with a declared schema, one transform that emits a computed field, and an output. Then run the full pre-flight before any real run:

1. clinker run mine.yaml --dry-run: does the config compile, and do the declared types check out?
2. clinker run mine.yaml --explain: does === DAG Topology === show the nodes wired the way you intended, and does === CXL Expressions === show the formula you meant?
3. Fix anything it surfaces, then run for real.

Predict, before you run: if you misspell a transform’s input:, both --dry-run and --explain reject the dangling reference during planning.

Retrieval checkpoint

✓ Checkpoint

// quick check

You set a schema field's `type:` to something that isn't a real type. Which command catches it fastest?

• --dry-run, since it validates declared schema types without reading data
• A full run over the whole file
• Nothing catches it until the output is wrong

// quick check

Carry forward from lesson 07: a pipeline passes `--dry-run` cleanly, but at run time a row's `amount.to_int()` fails under `strategy: continue`. What exit code do you get, and where does that bad row go?

• Exit code 2 (completed-with-DLQ); the bad row goes to the dead-letter queue
• Exit code 1; the job never started, so no row was processed
• Exit code 0; dry-run already proved the data was clean

Common mistakes

• Reading a clean --dry-run as “the pipeline is correct”. It only checks config structure, declared schema types, and the CXL compile, never your data; a bad row still fails at run time and lands in the DLQ (exit 2).
• Treating a successful plan as a data check. Wiring and CXL type errors are caught before execution; malformed input values still require an execution test.

Habit to build: run --dry-run after every edit, before any real run. It’s the cheapest bug-catch you have.

You should be able to:

• Run --dry-run and say what it validates (and what it skips)
• Read an --explain plan and trace a record through it
• Look up an E-code with explain --code

Verify in the checkout:

clinker run customer_etl.yaml --dry-run
clinker run customer_etl.yaml --explain
clinker explain --code E105