Read a plan with --explain

In Build & run Clinker you ran --explain and saw a few lines of output. Let’s actually read it. It’s the single best way to understand a pipeline before it runs, and your first glimpse of an idea that shapes the whole engine: Clinker plans a job before it runs it.

Prerequisites

• Navigate the workspace

→

Leads to

—

What you’ll be able to do

• Read an execution plan top to bottom and name what each line reports: DAG nodes, transform count, mode, arbitration.
• Name the four nodes of customer_etl’s DAG and say which kinds count as transforms and which don’t.
• Predict a plan from the YAML alone (its node count, its mode, its arbitration) before running --explain.
• Distinguish a change that reshapes the plan (a new node, a memory policy) from one that doesn’t (overriding a variable’s value).

New terms in this lesson (each is named here before it appears below):

• plan: the compiled, validated form of a job, printed by --explain.
• DAG, the directed, acyclic graph the nodes wire into.
• node: one step in that graph; only some kinds are transforms.
• tier, a dependency level in the source DAG (Tier 0 reads first).
• streaming mode: records flow straight through, not collected-then-emitted.
• arbitration, the memory policy the run would apply under pressure.

Predict first

You read the four-node customer_etl pipeline back in the build & run lesson. That’s enough to call its plan before you run a thing.

✋ Predict first

You're about to run `clinker run customer_etl.yaml --explain`. The pipeline is four nodes: one source, two transforms, one output, all wired in a line. Before you read on, predict three lines of the plan: how many DAG nodes? how many transforms? and what run mode, Streaming or something else?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

The plan counts what the YAML declared, by kind:

• DAG nodes: 4, one line per node: the source, two transforms, the output.
• Transforms: 2: only the two type: transform nodes. The source and output are nodes, but they are not transforms, so they don’t count here.
• Mode: Streaming, because each record can flow straight through; nothing in this pipeline has to collect every record before it can emit.

The trap is assuming --explain reads a slice of the data to produce these numbers. It reads zero records; the plan is built purely from the pipeline’s structure. That’s the whole point of the flag: it proves the job is well-formed (the nodes wire up, the CXL type-checks) before a byte moves. Hold those three numbers; the output below earns them.

--explain compiles, then stops

cd examples/pipelines
cargo run -p clinker -- run customer_etl.yaml --explain

--explain takes the YAML all the way through compilation (parsing, validation, typechecking the CXL, building the graph) and then prints the result instead of executing it. Nothing is read from the source; nothing is written. Here’s the heart of the output:

=== Resolved Outputs ===
  'results' → ./output/customers.csv

=== Execution Plan ===
Mode: Streaming
Transforms: 2
Output projections: 1
DAG nodes: 4
arbitration: BackPressurePreferred -> Priority

Source DAG:
  Tier 0: customers

Transform 'active_only':
  Parallelism: Stateless
Transform 'final_flag':
  Parallelism: Stateless

Read it top to bottom:

• Resolved Outputs: where results will land (./output/customers.csv).
• DAG nodes: 4, the source, two transforms, and the output, wired into a directed acyclic graph.
• Mode: Streaming, so records can flow straight through without the engine having to collect them all first.
• Source DAG / Tier 0: customers: the one source sits at tier 0, the first (and here only) wave of sources to read.
• Parallelism: Stateless (per transform): each row is independent, so these transforms have no cross-record state.
• arbitration: BackPressurePreferred -> Priority, the memory policy the run would use if it came under pressure. You’re seeing the engine’s bounded-memory machinery named, before a byte is allocated.

--explain even prints a Physical Properties section with each node’s predicted peak memory (for this tiny job, predicted_peak=345B, the size of the input CSV).

Why a separate plan at all?

That a plan exists as its own thing, printable, inspectable, produced before execution, is not an accident. Compilation turns the YAML into a typed, validated artifact, and only that artifact is handed to the runtime:

clinker-plan ·compiled.rs ·CompiledPlan type @19acdcb4

pub struct CompiledPlan {
    dag: ExecutionPlanDag,        // the lowered execution graph
    config: PipelineConfig,       // the validated configuration
    // …plus the validated compile outputs the runtime reads (typechecked CXL, bound schemas)
    // ...
}

We won’t unpack CompiledPlan yet; that’s a Planning & Expressions lesson. For now, just hold the shape of the idea: plan first, run second. --explain is you stepping in between the two.

Use → Modify → Create

You can read the plan as given. Now turn it into a skill: change one thing, predict how the plan shifts (or doesn’t), and verify. Each rung uses a real pipeline that ships in examples/pipelines/.

Use: read the real plan as given

You already did this above. Confirm you can answer, without re-running it, the three checkpoint-grade questions about customer_etl’s plan: how many DAG nodes (4), how many transforms (2), what mode (Streaming). If any of the three isn’t instant yet, re-read the output once more before moving on; the next two rungs build on reading this fluently.

Modify: change one thing, predict the new plan

Here’s the question that separates operating the flag from reading the engine: which changes reshape the plan, and which don’t? Two modifications, opposite answers.

Modify A: override a variable’s value. customer_etl declares a variable, gold_threshold (default 10000), and a channel can override it per tenant. There is no --var CLI flag; the override is a .channel.yaml overlay you pass with --channel. Acme’s channel sets the threshold to 50000:

channels/acme-corp/customer_etl.channel.yaml

channel:
  name: acme-corp
  target: ./customer_etl.yaml
vars:
  static:
    gold_threshold: { type: int, default: 50000 }

cargo run -p clinker -- run customer_etl.yaml \
  --channel channels/acme-corp/customer_etl.channel.yaml --explain

✋ Predict first

The channel raises gold_threshold from 10000 to 50000, the cutoff the tier expression compares against. Predict the plan: do DAG nodes, Transforms, Mode, or arbitration change versus the un-channeled run? Write down which lines (if any) move.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the expected plan

Nothing in the plan changes. It still reads DAG nodes: 4, Transforms: 2, Mode: Streaming, arbitration: BackPressurePreferred -> Priority, line for line identical to the un-channeled plan.

A variable override changes a value the run will compare against, not the shape of the job. The plan describes structure (which nodes exist, how they wire, how records flow, how memory is governed), and none of that depends on whether the gold cutoff is 10000 or 50000. (You’d see the difference in the per-record output, where fewer customers come out gold, exactly the value-vs-structure distinction Build & run Clinker drew between a reclassification and the run summary.)

Modify B: change the memory policy. Now a change that does move the plan. The scd_type2.yaml example declares a memory budget with a backpressure policy:

pipeline:
  name: scd_type2_backfill
  memory: { limit: "16K", backpressure: spill }

cargo run -p clinker -- run scd_type2.yaml --explain

✋ Predict first

customer_etl reported `arbitration: BackPressurePreferred -> Priority`. scd_type2 sets `backpressure: spill` in its memory config. Predict scd_type2's arbitration line: same as customer_etl, or different? (You don't need to predict the exact words; predict whether the line moves and why.)

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the expected plan

The arbitration line changes: scd_type2’s plan reads arbitration: Priority, not BackPressurePreferred -> Priority. Declaring backpressure: spill tells the engine to spill to disk under pressure rather than push back on its source, so the plan’s memory policy is reported differently.

This is the contrast that makes the skill real: a variable override (Modify A) leaves the plan untouched, but a memory-policy change rewrites a plan line. The plan is a mirror of structure and policy, not of data values. (scd_type2 also reads DAG nodes: 3 and Transforms: 0; its three nodes are a source, a reshape, and an output, and a reshape is not a transform, so the transform count is zero.)

Create: predict a plan from scratch, then verify

No worked answer this time. The tumbling_clicks.yaml example is a three-node pipeline: a source, an aggregate that buckets clicks into hourly windows, and an output. You have not seen its plan.

✋ Predict first

tumbling_clicks has three nodes: source → aggregate → output. From the node-counting rule you learned (every node counts as a DAG node; only `type: transform` nodes count as transforms), predict its plan's `DAG nodes` line and its `Transforms` line BEFORE running. Then run `clinker run tumbling_clicks.yaml --explain` and check.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer, then verify

DAG nodes: 3 and Transforms: 0.

DAG nodes: 3 because every node counts: source + aggregate + output. Transforms: 0 because none of the three is a type: transform node: a source isn’t, an output isn’t, and crucially an aggregate isn’t a transform either. The transform count only counts type: transform nodes, so a pipeline can do real per-record work in an aggregate and still report Transforms: 0.

Verify it:

cargo run -p clinker -- run tumbling_clicks.yaml --explain

Mode: Streaming
Transforms: 0
DAG nodes: 3
arbitration: BackPressurePreferred -> Priority

Source DAG:
  Tier 0: clicks

If you predicted Transforms: 1 (counting the aggregate as a transform) or Transforms: 3 (counting every node), the real plan corrects you on the spot: that’s the gap between “a node” and “a transform” closing.

Comfortable reading plans across all three example pipelines already? Skip ahead →

Retrieval checkpoint

✓ Checkpoint

// quick check

What does `--explain` actually do?

• Runs the pipeline and prints extra logging
• Compiles the pipeline to a plan and prints it, without executing
• Checks the YAML syntax only, without building the graph

// quick check

customer_etl's plan prints `DAG nodes: 4` and `Transforms: 2`. tumbling_clicks's prints `DAG nodes: 3` and `Transforms: 0`. Which statement is the MISCONCEPTION to correct?

• `DAG nodes` counts every node; `Transforms` counts only `type: transform` nodes, so an aggregate raises DAG nodes but not the transform count
• An aggregate node does per-record work, so it must be counted in the `Transforms` line
• A source and an output are DAG nodes but are not transforms

// quick check

You run `--explain` on customer_etl with `--channel ...acme...` which overrides gold_threshold from 10000 to 50000. What changes in the printed plan?

• `Transforms` drops, because fewer customers will be flagged `gold`
• Nothing: the plan describes structure and policy, and a variable's value is neither; the same 4 nodes wire the same way
• `Mode` switches from Streaming to a buffered mode to apply the new threshold

// quick check

Spaced recall from build & run: you want to confirm the example produces the right OUTPUT, not just that it compiles to a valid plan. Which flag do you reach for, and why?

• `--explain`, because it runs the pipeline and shows the output records
• `--dry-run`, because it actually processes records (writing the result to your terminal), whereas `--explain` moves no data at all
• Either one, since both run the pipeline and print the resulting rows

You should be able to:

• You can name the four DAG nodes in customer_etl's plan, and which two are transforms
• You explained, in a sentence, what `--explain` does and does not do
• You predicted that overriding gold_threshold via a channel leaves the plan unchanged
• You predicted tumbling_clicks's plan (3 DAG nodes, 0 transforms) and verified it

Verify in the checkout:

cd examples/pipelines
cargo run -p clinker -- run customer_etl.yaml --explain
cargo run -p clinker -- run customer_etl.yaml --channel channels/acme-corp/customer_etl.channel.yaml --explain
cargo run -p clinker -- run tumbling_clicks.yaml --explain

That’s Orientation. You can build the engine, run a pipeline, work the compiler loop, navigate the crates, and read a plan, and predict one from the YAML before you run it. Next comes A Record’s Journey, where we follow one record all the way through that four-node DAG and start opening the doors.

From the pipeline author’s side (optional, one-directional; the same --explain plan read from the pipeline author’s side, no engine internals):

Want to read plans from the YAML author's side, not the engine builder's? Skip ahead →

Mark this lesson complete