From YAML to a plan

You wrote customer_etl.yaml: a few nodes and some CXL. But YAML is just text. In reading a plan with --explain, --explain printed an execution plan instead. Something turned your text into that. This lesson follows that transformation, shallowly. The deep version is Planning & Expressions. You’ll build a miniature of the transformation in code so you can predict its shape rather than memorize it.

Prerequisites

• What is a record?

→

Leads to

• Through the DAG

What you’ll be able to do

• Name the steps that turn YAML into a runnable plan, in order.
• Trace a single value from source text to the plan, and say what Spanned adds along the way.
• Predict which stage rejects each kind of mistake: a cycle, an unknown column, a nonsense expression.
• Explain why Clinker compiles to a CompiledPlan before it runs, in terms of what the runtime is handed.

New terms in this lesson (each is also expanded inline at the point you first need it):

• node: one step in the pipeline graph; a type: in your nodes: list.
• DAG: a directed acyclic graph: nodes wired by edges, with no path that loops back on itself. The plan’s nodes form one.
• validation: the stage that checks the config is well-formed, before any data moves: no cycles, every input exists, paths are allowed.
• Spanned: a wrapper that pairs a parsed value with where it came from in the source (line & column), so an error can point at the exact spot.

Predict first

✋ Predict first

Your YAML has a transform whose CXL reads a column status_clean that no upstream node ever produces. You run --explain. Does the engine (a) print a plan that quietly omits that column, (b) refuse and report an error, or (c) build the plan and only fail later when records actually flow? Write down your pick and the reason before revealing.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

The answer is (b): it refuses, at plan time, before a single record moves.

Referencing a column that no node produces is exactly the kind of mistake the validation stage exists to catch. The whole point of compiling YAML to a plan first is that errors like this surface before execution. You find out at --explain time, not halfway through a million-row file with a half-written output.

The tempting wrong model is (c): “it’ll build something and blow up at runtime.” That’s how a system without a planning phase behaves: it discovers the broken reference only when a record arrives. Clinker is built the opposite way. The runtime is never handed raw YAML, only a CompiledPlan that, by existing, proves the references resolve. Hold that contrast; the rest of the lesson earns it.

The journey from text to plan

Before a single record moves, your YAML goes through a pipeline of its own:

YAML text
   │  parse (span-aware — every value remembers its line & column)
   ▼
config  (typed nodes, still just "what you asked for")
   │  validate  (no cycles? all inputs exist? paths allowed?)
   ▼
   │  typecheck the CXL  (does `lifetime_value.to_int()` make sense?)
   ▼
   │  lower to a graph
   ▼
CompiledPlan  ──►  handed to the runtime

Each arrow is a stage that can reject your pipeline, and each rejects a different class of mistake:

• parse catches malformed YAML: a bad indent, an unterminated string.
• validate catches structural mistakes: a cycle in the graph, a node that reads an input no other node produces, a path you’re not allowed to write.
• typecheck catches nonsense expressions, like to_int() on something that can never be an integer.
• lower turns the validated config into the execution graph the runtime understands.

The parse step is span-aware: each parsed value carries its source location, so a mistake can be reported with a precise line and column rather than a vague “something’s wrong.” That location-carrying wrapper is Spanned:

clinker-plan ·yaml.rs ·Spanned type @19acdcb4

pub struct Spanned<T> {
    pub value: T,
    pub referenced: Location,   // where this value is used in the source
    pub defined: Location,      // where it was defined (e.g. a YAML anchor)
}

The output is a CompiledPlan

Everything upstream converges on one typed artifact, the CompiledPlan. It bundles the lowered graph, the validated config, and the typechecked CXL:

clinker-plan ·compiled.rs ·CompiledPlan type @19acdcb4

pub struct CompiledPlan {
    dag: ExecutionPlanDag,        // the lowered execution graph
    config: PipelineConfig,       // the validated configuration
    // …plus the validated compile outputs the runtime reads (typechecked CXL, bound schemas)
    // ...
}

When you ran --explain, you saw the engine do all of this (parse, validate, typecheck, lower) and then print the result instead of executing. The execution plan you read was this CompiledPlan, rendered as text. Its dag field holds the nodes you counted in that output (DAG nodes: 4), now wired into a graph with no cycles: a DAG.

Why plan first?

Two reasons, both of which you can already feel:

1. Errors surface before any data moves. If a column name is misspelled or a CXL expression is nonsense, you find out at plan time, not halfway through a million-row file with a half-written output.
2. The runtime gets a proof, not a wish. The executor never receives raw YAML. It only ever accepts a CompiledPlan, an artifact that, by existing, proves the pipeline is well-formed. The boundary between “planning” and “running” is one of Clinker’s defining design decisions; we open it up in Planning & Expressions.

On the Rust side, this whole stage runs on Result: parsing, validation, and typechecking each return success or a typed error rather than throwing. Errors are values you handle, and they roll up into one error vocabulary (PipelineError, covered in the error-handling lesson).

Build the pipeline in miniature

You’ll understand the transformation best by building a small version of it. Each rung below is a self-contained, runnable Rust program. We stage one new idea at a time: first a span that travels with a value, then the stages as fallible steps, then the whole compile as a function you write.

Worked: a value that remembers where it came from

The first thing parse does is wrap every value in its source location, so a later stage can blame the right line. Here is Spanned in miniature: a value paired with a Location, and a parse step that attaches one. Run it and watch the location ride along.

rust // editable

reset ▶ run

// A tiny stand-in for the engine's Location: file line & column.
#[derive(Debug, Clone, Copy)]
struct Location { line: u32, col: u32 }

// The engine's wrapper, trimmed: a value plus where it was used.
#[derive(Debug)]
struct Spanned<T> {
  value: T,
  referenced: Location, // where this value appears in the source
}

// "Parsing" one entry: keep the value AND remember its line.
fn parse_entry(value: &str, line: u32) -> Spanned<String> {
  Spanned { value: value.to_string(), referenced: Location { line, col: 1 } }
}

fn main() {
  // Pretend these are two node names read from a YAML file.
  let parsed = vec![
      parse_entry("customers", 3),
      parse_entry("active_only", 7),
  ];

  for entry in &parsed {
      // Because the span travels with the value, any later error
      // can say EXACTLY where to look.
      println!("node {:?} declared at line {}", entry.value, entry.referenced.line);
  }
}

// A tiny stand-in for the engine's Location: file line & column. #[derive(Debug, Clone, Copy)] struct Location { line: u32, col: u32 } // The engine's wrapper, trimmed: a value plus where it was used. #[derive(Debug)] struct Spanned<T> { value: T, referenced: Location, // where this value appears in the source } // "Parsing" one entry: keep the value AND remember its line. fn parse_entry(value: &str, line: u32) -> Spanned<String> { Spanned { value: value.to_string(), referenced: Location { line, col: 1 } } } fn main() { // Pretend these are two node names read from a YAML file. let parsed = vec![ parse_entry("customers", 3), parse_entry("active_only", 7), ]; for entry in &parsed { // Because the span travels with the value, any later error // can say EXACTLY where to look. println!("node {:?} declared at line {}", entry.value, entry.referenced.line); } }

The takeaway: the span is not metadata stored off to one side. It is glued to the value by the type. Every stage downstream still has it, so a validation error three steps later can still point at line 7.

Completion: validate the graph

Now the validation stage. Below, a config is a list of nodes, each naming the input it reads. Validation must reject a node whose input no other node produces, the exact mistake from the Predict-first opener. Two checks are written; complete the third arm that flags an unknown input.

rust // editable

reset ▶ run

#[derive(Debug)]
struct Node {
  name: &'static str,
  reads: Option<&'static str>, // the node this one consumes, or None for a source
}

// Validation returns a Result: Ok(()) if the graph is sound, Err(msg) if not.
fn validate(nodes: &[Node]) -> Result<(), String> {
  for node in nodes {
      match node.reads {
          None => {} // a source node reads nothing upstream — fine
          Some(input) => {
              let produced = nodes.iter().any(|n| n.name == input);
              if !produced {
                  // TODO: this input names a node that does not exist.
                  // Return an Err that says which node and which missing input.
                  todo!()
              }
          }
      }
  }
  Ok(())
}

fn main() {
  let good = [
      Node { name: "customers", reads: None },
      Node { name: "active_only", reads: Some("customers") },
  ];
  let broken = [
      Node { name: "customers", reads: None },
      Node { name: "final_flag", reads: Some("status_clean") }, // no such node
  ];

  println!("good:   {:?}", validate(&good));
  println!("broken: {:?}", validate(&broken));
}

#[derive(Debug)] struct Node { name: &'static str, reads: Option<&'static str>, // the node this one consumes, or None for a source } // Validation returns a Result: Ok(()) if the graph is sound, Err(msg) if not. fn validate(nodes: &[Node]) -> Result<(), String> { for node in nodes { match node.reads { None => {} // a source node reads nothing upstream — fine Some(input) => { let produced = nodes.iter().any(|n| n.name == input); if !produced { // TODO: this input names a node that does not exist. // Return an Err that says which node and which missing input. todo!() } } } } Ok(()) } fn main() { let good = [ Node { name: "customers", reads: None }, Node { name: "active_only", reads: Some("customers") }, ]; let broken = [ Node { name: "customers", reads: None }, Node { name: "final_flag", reads: Some("status_clean") }, // no such node ]; println!("good: {:?}", validate(&good)); println!("broken: {:?}", validate(&broken)); }

💡 Hint 1

You’re inside the if !produced branch, so you already know the input is missing. Return an Err(String) naming both node.name and input. That’s the precise message a span would let the real engine attach to a line.

Show solution

return Err(format!(
    "node {:?} reads {:?}, but no node produces it",
    node.name, input,
));

This is validation in one line of judgment: walk the declared graph, and the moment a node reads something nothing produces, refuse before a record exists. The real engine reports the same class of error, plus the line number that the Spanned wrapper carried this far.

Faded: compile the whole thing

Your turn with the stages wired together. Write compile, which runs the journey from the diagram: parse → validate → lower, returning Result<CompiledPlan, String>. Each helper already returns a Result; chain them so that any stage failing aborts the compile with its error, and only an all-green run produces a plan. (The ? operator does exactly this early-return-on-Err; if it’s new, the match form works too.)

rust // editable

reset ▶ run

#[derive(Debug)]
struct CompiledPlan { node_count: usize }

// Each stage already returns a Result; sequence them in order.
fn parse(src: &str) -> Result<Vec<&str>, String> {
  if src.is_empty() { return Err("empty source".to_string()); }
  Ok(src.split(',').collect())
}

fn validate(nodes: &[&str]) -> Result<(), String> {
  if nodes.iter().any(|n| n.is_empty()) {
      return Err("a node has no name".to_string());
  }
  Ok(())
}

fn lower(nodes: &[&str]) -> CompiledPlan {
  CompiledPlan { node_count: nodes.len() }
}

// TODO: run parse -> validate -> lower. If parse or validate returns Err,
// return that Err from compile. Only on success, return Ok(lower(...)).
fn compile(src: &str) -> Result<CompiledPlan, String> {
  // your code here
  todo!()
}

fn main() {
  println!("{:?}", compile("customers,active_only,final_flag"));
  println!("{:?}", compile(""));            // parse rejects
  println!("{:?}", compile("customers,,x")); // validate rejects
}

#[derive(Debug)] struct CompiledPlan { node_count: usize } // Each stage already returns a Result; sequence them in order. fn parse(src: &str) -> Result<Vec<&str>, String> { if src.is_empty() { return Err("empty source".to_string()); } Ok(src.split(',').collect()) } fn validate(nodes: &[&str]) -> Result<(), String> { if nodes.iter().any(|n| n.is_empty()) { return Err("a node has no name".to_string()); } Ok(()) } fn lower(nodes: &[&str]) -> CompiledPlan { CompiledPlan { node_count: nodes.len() } } // TODO: run parse -> validate -> lower. If parse or validate returns Err, // return that Err from compile. Only on success, return Ok(lower(...)). fn compile(src: &str) -> Result<CompiledPlan, String> { // your code here todo!() } fn main() { println!("{:?}", compile("customers,active_only,final_flag")); println!("{:?}", compile("")); // parse rejects println!("{:?}", compile("customers,,x")); // validate rejects }

Show solution

fn compile(src: &str) -> Result<CompiledPlan, String> {
    let nodes = parse(src)?;   // parse fails -> return its Err here
    validate(&nodes)?;         // validate fails -> return its Err here
    Ok(lower(&nodes))          // both passed -> lower can't fail; build the plan
}

That’s the engine’s spine in three lines: each stage is a fallible Result-returning step, ? short-circuits on the first failure, and a CompiledPlan is produced only when every stage succeeded. The real pipeline adds a typecheck stage and far richer types, but the shape (sequence fallible stages, emit a proof at the end) is exactly this.

What the compiler enforces

Read the cited CompiledPlan again with the ladder behind you. Notice what its existence guarantees: the runtime’s entry point takes a CompiledPlan, not a string of YAML. So a caller cannot hand the executor an unvalidated pipeline. There is no constructor for CompiledPlan that skips the stages. The type is the proof.

What a junior might misread: thinking --explain is “just a linter” that scans syntax. It runs the entire compile (parse, validate, typecheck, lower), and the plan it prints is the genuine CompiledPlan the runtime would consume. The only thing it skips is execution.

Want Result and the ? operator drilled from first principles in The Rust Book? Read the docs →

Why-bridge: the plan is a boundary, not a formality

Why does the engine bother producing a separate, typed CompiledPlan at all, instead of interpreting the YAML as it goes? Because the plan is a boundary: everything that can be checked statically is checked once, up front, and the result is an artifact that carries that guarantee forward. The runtime never re-validates, never re-parses; it trusts the CompiledPlan because nothing else can construct one.

That is why a broken column reference fails at --explain time and not at row one million: the failure happens in the stage that can see the whole graph, before the boundary, while there’s still nothing to clean up.

Retrieval checkpoint

✓ Checkpoint

// quick check

Your YAML names a transform that reads a column no upstream node produces. When do you find out?

• At runtime, once records start flowing and the column comes up missing
• At plan time: the validate stage rejects it before any data moves
• Never; the engine silently drops the unknown column

// quick check

What does the Spanned<T> wrapper add to a parsed value?

• It makes the value faster to copy
• It pairs the value with its source location (line & column) so a later error can point at the exact spot
• It marks the value as already validated

// quick check

MISCONCEPTION check: which statement about --explain is wrong and needs correcting?

• It runs the full compile (parse, validate, typecheck, lower), then prints the plan instead of executing
• It is just a YAML syntax linter; it does not build the graph or check references
• The plan it prints is the same CompiledPlan the runtime would consume

// quick check

Spaced recall from reading a plan with --explain: you ran customer_etl.yaml --explain and saw 'DAG nodes: 4', then 'Mode: Streaming'. Was any data read from the source to produce that output?

• Yes, it reads the source once to count the records before planning
• No: --explain compiles and prints the plan instead of executing; no source is read and nothing is written
• Only the first row, to infer the schema

You should be able to:

• You can list the stages in order: parse → validate → typecheck CXL → lower → CompiledPlan
• You can say which stage rejects a cycle, an unknown input, and a nonsense expression
• You can explain what Spanned adds and why a value carries its source location
• You wrote a compile function that sequences fallible stages and emits a plan only on success
• You connected --explain's output to the CompiledPlan it actually builds

Verify in the checkout:

cd examples/pipelines
cargo run -p clinker -- run customer_etl.yaml --explain

Connections

You can now name the stages that turn YAML into a plan, trace a value through them with its span, predict which stage rejects which mistake, and build a miniature compile that emits a plan only when every stage passes. Next: how a record actually travels through the graph that plan describes.

Prerequisites

• What is a record?

→

Leads to

• Through the DAG

Go deeper on the Rust (optional, one-directional; the same Result/? mechanics taught from first principles in The Rust Book):

• Want Result and the ? operator drilled on a standalone parser? Read the docs →

Glossary terms used: node, Result, record.

Mark this lesson complete