Validate before you run

The question for this lesson: you’ve edited a 200-line pipeline. Before you run it over a million rows, how do you check it’s correct, without waiting for the whole run?

What you’ll be able to do

• Use --dry-run to validate config structure and declared types without processing data, and know what it leaves for the real run.
• Use --explain to print the execution plan and read it.
• Decode a diagnostic E-code with explain --code.

Check without running: --dry-run

clinker run customer_etl.yaml --dry-run

--dry-run parses your YAML, checks the config’s structure, validates your declared schema types, and resolves the output paths, then stops, without reading a single data row. If a required field is missing or a schema type doesn’t exist, you find out in a second.

Be honest about its reach, though: --dry-run validates the structure and declared types before reading data. Deeper logic errors (a dangling input: that names a node which isn’t there, an unknown field or method inside a CXL expression) aren’t caught here. Those surface when you actually run, because they depend on the data flowing through. So treat --dry-run as the fast first gate, not a full proof of correctness.

🌱 New here? — why it's instant

Validation doesn’t touch your data. Clinker checks the shape of the job (the config structure, the declared schema types, the resolved outputs), which is fast no matter how big the file is. Only a real run reads rows. So --dry-run costs the same whether your file has 10 rows or 10 million.

You can also process just the first few rows to sanity-check output: --dry-run -n 5 runs 5 records to stdout instead of the output file.

See the shape: --explain

clinker run customer_etl.yaml --explain

--explain prints the execution plan in labelled sections (the CXL it compiled, the node graph, and how data flows), so you can confirm the pipeline does what you think before you run it. The full output is long; here are the three sections you’ll read most (trimmed):

=== Execution Plan ===

Mode: Streaming
Transforms: 1
Output projections: 1
DAG nodes: 3

=== CXL Expressions ===

Transform 'tagged':
  emit is_active = status == "active"

=== DAG Topology ===

  ● [source] customers
  │ [transform] tagged
  │ [output] results

✋ Predict first

Read the DAG Topology section. A record enters at the source. Through which nodes does it pass on its way to a file?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

customers (the source) → tagged (the transform) → results (the output). The === DAG Topology === section lists the nodes top-to-bottom in flow order, so you read it straight down. The === CXL Expressions === section above it shows the exact expression Clinker compiled for each transform. Reading both is how you confirm the wiring and the logic match your intent before spending a full run.

Decode a diagnostic: explain --code

When a run or dry-run fails, it prints an E-code. You don’t have to guess what it means; look any code up directly, with no config needed:

clinker explain --code E105

That prints a full explanation of the diagnostic: what it means, an example that triggers it, and how to fix it. E105, for instance, is “Channel binding references undeclared config key”: a channel file bound a config key the target pipeline never declared. The point isn’t to memorize codes. Whenever a run prints one, explain --code turns it into a plain explanation on the spot.

Use: validate a pipeline

Run --dry-run on this one. Predict whether it passes, then check.

✋ Predict first

Read every node name and every input. Does each input match a real node name? Will --dry-run pass or report an error?

Reveal the expected result

=== Resolved Outputs ===

'results' → ./output/customers.csv
    [if_exists=overwrite, write_meta=false]

Dry run: config valid, 1 inputs, 1 outputs, 1 transforms

pipeline.yaml // editable

reset ⌖ explain ✓ dry-run ▶ run

pipeline: name: customer_etl nodes: - type: source name: customers config: name: customers type: csv path: ./data/customers.csv options: { has_header: true } schema: - { name: status, type: string } - type: transform name: tagged input: customers config: cxl: | emit is_active = status == "active" - type: output name: results input: tagged config: name: results type: csv path: ./output/customers.csv

Modify: break it on purpose

Now make --dry-run fail, and find the one break it lets through. In the runner above, try two changes, one at a time, and predict the outcome of each before you re-run.

✋ Predict first

In the pipeline, (a) change a schema field's `type:` to a made-up type like `wizbang`, then dry-run. Then undo that and (b) misspell a transform's `input:` so it names a node that isn't there, and dry-run. For each, does --dry-run catch it, and what exit code do you expect?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

(a) The made-up type: is a declared-type problem, so --dry-run catches it on the spot: the config never compiles and you get exit code 1 (compile error, the job never starts). (b) The misspelled input: is a wiring problem: it depends on how data flows between nodes, which dry-run doesn’t trace. So --dry-run reports config valid and passes, and the dangling input only surfaces on a real run. That gap is the whole point of the next two sections: a clean dry-run narrows the risk, it doesn’t remove it.

// quick check

What does `--dry-run` validate before reading any data?

• Config structure and declared schema types, and it resolves the output paths
• Every CXL expression's field and method references, end to end
• Nothing; it just prints the YAML back

Create: pre-flight a pipeline you build

Now put the tools together on a pipeline of your own. Author a small one from scratch: a csv source with a declared schema, one transform that emits a computed field, and an output. Then run the full pre-flight before any real run:

1. clinker run mine.yaml --dry-run: does the config compile, and do the declared types check out?
2. clinker run mine.yaml --explain: does === DAG Topology === show the nodes wired the way you intended, and does === CXL Expressions === show the formula you meant?
3. Fix anything it surfaces, then run for real.

Predict, before you run: if you misspell a transform’s input:, which step catches it? Not --dry-run. A dangling input: is a wiring error, not a declared-type error, so it passes dry-run. Reading the --explain topology is what catches it before you waste a full run.

Retrieval checkpoint

✓ Checkpoint

// quick check

You set a schema field's `type:` to something that isn't a real type. Which command catches it fastest?

• --dry-run, since it validates declared schema types without reading data
• A full run over the whole file
• Nothing catches it until the output is wrong

// quick check

Carry forward from lesson 07: a pipeline passes `--dry-run` cleanly, but at run time a row's `amount.to_int()` fails under `strategy: continue`. What exit code do you get, and where does that bad row go?

• Exit code 2 (completed-with-DLQ); the bad row goes to the dead-letter queue
• Exit code 1; the job never started, so no row was processed
• Exit code 0; dry-run already proved the data was clean

Common mistakes

• Reading a clean --dry-run as “the pipeline is correct”. It only checks config structure, declared schema types, and the CXL compile, never your data; a bad row still fails at run time and lands in the DLQ (exit 2).
• Expecting --dry-run to catch a misspelled input:. Wiring and logic errors depend on data flowing between nodes, so they surface only on a real run; dry-run validates config, not data flow.

Habit to build: run --dry-run after every edit, before any real run. It’s the cheapest bug-catch you have.

You should be able to:

• Run --dry-run and say what it validates (and what it skips)
• Read an --explain plan and trace a record through it
• Look up an E-code with explain --code

Verify in the checkout:

clinker run customer_etl.yaml --dry-run
clinker run customer_etl.yaml --explain
clinker explain --code E105