The plan/runtime boundary

The executor is the engine’s hot, parallel, memory-bounded core, the part you least want running a half-checked plan. The last three lessons built up the validated pieces: a screened path, a strict located parse. This lesson is where they converge into a single typed handle, CompiledPlan, and where the engine makes “this plan is fully compiled” the same kind of unforgeable, compiler-checked fact that ValidatedPath made of “this path is screened.”

Prerequisites

• Span-preserving parse & strictness

→

Leads to

• Error handling

What you’ll be able to do

• Name what CompiledPlan freezes in (the lowered DAG, the typechecked CXL, the bound schemas, the source hash) and say why each is settled once.
• Distinguish plan-time (static, done once) work from run-time (dynamic, per record) work, and give two examples of each.
• Explain why run_plan_with_readers_writers accepts &CompiledPlan and refuses a raw &PipelineConfig, in terms of the only constructor.
• Predict the compiler error you get when you hand the executor an unvalidated plan, and trace why the type is the proof that compilation happened.

New terms in this lesson (each is also expanded inline at the point you first need it):

• plan (CompiledPlan): the compiled, validated form of a job; here, the typed handle the runtime consumes.
• runtime / executor: the per-record engine that runs a plan; it accepts only the handle.
• compile-time validation: a guarantee enforced by the type system, so a violation fails to build, not at run time.
• the boundary: the membrane CompiledPlan draws between decide once and run many.

Predict first

✋ Predict first

The executor's entry point is fn run_plan_with_readers_writers(plan: &CompiledPlan, ...). You have a PipelineConfig (the validated config) in hand but you have NOT called compile. You try to pass it where a &CompiledPlan is expected. Does the program (a) compile and run, silently re-validating at run time, (b) compile but panic on the first record, or (c) fail to compile at all? Write down your pick and the reason before revealing.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

The answer is (c): it does not compile. A PipelineConfig is not a CompiledPlan, and you cannot hand-build a CompiledPlan to bridge the gap: its fields are private and its only constructor is PipelineConfig::compile. So “run an uncompiled plan” is not a bug you catch at run time; it is a sentence the type system refuses to let you write.

The tempting wrong model is (a): “the executor will re-validate whatever it’s given.” That is how an engine without a boundary behaves, re-checking on every entry because it can’t trust its input. Clinker is built the opposite way: the &CompiledPlan parameter is the proof that compilation already happened, so the run-time path never re-validates. Hold that contrast; the rest of the lesson earns it, and a compile_fail doctest in the real source pins exactly this refusal.

A handle that means “compiled”

CompiledPlan is the output of compilation, and like ValidatedPath, its fields are private, so the only way to hold one is to have gone through the compiler:

clinker-plan ·compiled.rs ·CompiledPlan type @19acdcb4

/// CompiledPlan is the typed-handle output of PipelineConfig::compile.
#[derive(Debug)]
pub struct CompiledPlan {
    dag: ExecutionPlanDag,             // the lowered execution DAG
    config: PipelineConfig,            // the validated config
    // …plus the validated compile outputs the runtime reads (typechecked CXL, bound schemas)
    channel_identity: Option<ChannelIdentity>,
    pipeline_hash: [u8; 32],           // BLAKE3 of the source YAML
}

Everything expensive and checkable has already happened by the time this struct exists. The DAG is lowered (nodes resolved, topology fixed). Every CXL expression is typechecked, its typed program frozen into the plan. Schemas are bound. The source is hashed. The struct is read-only from the outside: accessors like dag() and config() hand out shared references, nothing more.

The static/runtime split: decide once, run many

This is the architectural heart of the lesson. Work in a streaming engine falls into two buckets:

• Plan-time (static), done once per job: parse, validate, resolve references, lower the DAG, typecheck every CXL expression, bind schemas. Expensive, but paid a single time.
• Run-time (dynamic), done once per record: pull a record, evaluate the already-typechecked expressions, route it, write it. This runs millions of times.

CompiledPlan is the membrane between them. Anything that can be settled before the first record flows is settled at compile time and frozen into the handle, so the per-record path never repeats it. Each transform’s Arc<TypedProgram> is frozen into the plan, and the executor reads it directly instead of re-typechecking. The executor never typechecks a CXL expression at run time. That was done once, at compile time, and the result is in the handle. (CXL: a staged language shows the same “compile once, run per record” idea inside CXL itself.)

The nodes the DAG is built from are themselves self-describing: each PlanNode owns its topology, its display form, and its execution payload, plus a span back to the source YAML for diagnostics:

clinker-plan ·mod.rs ·PlanNode type @19acdcb4

pub enum PlanNode {
    Source { name: String, /* ... */ },
    Transform { name: String, /* ... */ },
    Output { name: String, /* ... */ },
    // ... each variant owns its topology, display, and execution payload
}

The executor accepts the handle and nothing else

Here’s the enforcement. The executor’s public entry point takes &CompiledPlan. It does not accept a PipelineConfig. And a compile_fail doctest pins that refusal so it can never silently regress:

clinker-exec ·mod.rs ·run_plan_with_readers_writers fn @19acdcb4

/// Accepts the typed `CompiledPlan` handle returned by `PipelineConfig::compile`.
///
/// ```compile_fail
///     let _ = PipelineExecutor::run_plan_with_readers_writers(
///         cfg, // ← should be &CompiledPlan, not &PipelineConfig
///         // ...
///     );
/// ```
pub fn run_plan_with_readers_writers<W: Into<WriterRegistry>>(
    plan: &CompiledPlan,
    // ...
) { /* ... */ }

Trace the guarantee, exactly as with ValidatedPath: to run, you need a &CompiledPlan; to get a CompiledPlan, you must call PipelineConfig::compile (the only constructor); and compile is where all validation lives. So “the executor ran an unvalidated plan” is not a bug you guard against at run time. It’s a program that does not compile. The type is the proof that compilation happened.

Build the boundary yourself

The whole pattern fits on one screen, and you’ll build it in three rungs: first read a fully worked miniature, then complete the decide-once step, then write the boundary from a skeleton. Each rung is a self-contained, runnable Rust program. One new idea per rung.

Worked: the boundary, end to end

Here is the pattern complete and annotated: a raw plan, a compile step that validates and bakes a result, a private-fielded handle, and a runner that accepts only the handle. Run it, then read the commented-out block to see the refusal.

rust // editable

reset ▶ run

mod plan {
  pub struct RawPlan { pub steps: Vec<String> }

  // Typed handle — private fields, so only `compile` can build one.
  pub struct CompiledPlan {
      steps: Vec<String>,
      step_count: usize, // computed ONCE at compile time
  }

  impl CompiledPlan {
      pub fn step_count(&self) -> usize { self.step_count }
      pub fn steps(&self) -> &[String] { &self.steps }
  }

  // The ONLY constructor: validate + lower once, freeze into the handle.
  pub fn compile(raw: RawPlan) -> Result<CompiledPlan, String> {
      if raw.steps.is_empty() {
          return Err("a plan needs at least one step".into());
      }
      let step_count = raw.steps.len();
      Ok(CompiledPlan { steps: raw.steps, step_count })
  }
}

// The executor accepts ONLY the compiled handle.
fn run(plan: &plan::CompiledPlan) {
  println!("running {} steps: {:?}", plan.step_count(), plan.steps());
}

fn main() {
  let raw = plan::RawPlan { steps: vec!["read".into(), "filter".into(), "write".into()] };
  match plan::compile(raw) {
      Ok(compiled) => run(&compiled),
      Err(e) => println!("compile error: {e}"),
  }

  // Try this: skip compilation and hand the runner a raw plan.
  // let raw2 = plan::RawPlan { steps: vec!["read".into()] };
  // run(&raw2);
  // ^ won't compile: run() wants &CompiledPlan, and you can't forge one —
  //   its fields are private; only plan::compile produces one.
}

mod plan { pub struct RawPlan { pub steps: Vec<String> } // Typed handle — private fields, so only `compile` can build one. pub struct CompiledPlan { steps: Vec<String>, step_count: usize, // computed ONCE at compile time } impl CompiledPlan { pub fn step_count(&self) -> usize { self.step_count } pub fn steps(&self) -> &[String] { &self.steps } } // The ONLY constructor: validate + lower once, freeze into the handle. pub fn compile(raw: RawPlan) -> Result<CompiledPlan, String> { if raw.steps.is_empty() { return Err("a plan needs at least one step".into()); } let step_count = raw.steps.len(); Ok(CompiledPlan { steps: raw.steps, step_count }) } } // The executor accepts ONLY the compiled handle. fn run(plan: &plan::CompiledPlan) { println!("running {} steps: {:?}", plan.step_count(), plan.steps()); } fn main() { let raw = plan::RawPlan { steps: vec!["read".into(), "filter".into(), "write".into()] }; match plan::compile(raw) { Ok(compiled) => run(&compiled), Err(e) => println!("compile error: {e}"), } // Try this: skip compilation and hand the runner a raw plan. // let raw2 = plan::RawPlan { steps: vec!["read".into()] }; // run(&raw2); // ^ won't compile: run() wants &CompiledPlan, and you can't forge one — // its fields are private; only plan::compile produces one. }

Uncomment the last block and the build fails: run wants a &CompiledPlan, you have a RawPlan, and you cannot hand-build a CompiledPlan because its fields are private. The only way forward is through compile, which is the only place validation happens. That refusal is the plan/runtime boundary in one compiler error.

Completion: decide once at the boundary

The new idea this rung isolates is the decide-once property, the whole point of the boundary. Below, compile validates, but it does not yet pre-compute the expensive thing that the per-record runner would otherwise redo on every record. The runner here loops over records and, for each one, needs the uppercased step names. Computing those uppercases is plan-time work: it depends only on the plan, not on the record, so it must be settled once inside compile and frozen into the handle, never recomputed in the hot loop. Fill the gap so the handle carries the pre-computed names.

rust // editable

reset ▶ run

mod plan {
  pub struct RawPlan { pub steps: Vec<String> }

  pub struct CompiledPlan {
      // The runner reads THIS instead of recomputing per record.
      upper_steps: Vec<String>,
  }

  impl CompiledPlan {
      pub fn upper_steps(&self) -> &[String] { &self.upper_steps }
  }

  pub fn compile(raw: RawPlan) -> Result<CompiledPlan, String> {
      if raw.steps.is_empty() {
          return Err("a plan needs at least one step".into());
      }
      // TODO: do the uppercasing ONCE here, at plan time, and store it.
      // Build upper_steps from raw.steps so the per-record loop never redoes it.
      let upper_steps: Vec<String> = todo!();
      Ok(CompiledPlan { upper_steps })
  }
}

// Per-record runtime path — runs MANY times. It must only READ the frozen work.
fn run(plan: &plan::CompiledPlan, records: &[u32]) {
  for r in records {
      // Reads the pre-computed names; does NOT uppercase here.
      println!("record {r}: {:?}", plan.upper_steps());
  }
}

fn main() {
  let raw = plan::RawPlan { steps: vec!["read".into(), "filter".into(), "write".into()] };
  match plan::compile(raw) {
      Ok(compiled) => run(&compiled, &[1, 2, 3]),
      Err(e) => println!("compile error: {e}"),
  }
}

mod plan { pub struct RawPlan { pub steps: Vec<String> } pub struct CompiledPlan { // The runner reads THIS instead of recomputing per record. upper_steps: Vec<String>, } impl CompiledPlan { pub fn upper_steps(&self) -> &[String] { &self.upper_steps } } pub fn compile(raw: RawPlan) -> Result<CompiledPlan, String> { if raw.steps.is_empty() { return Err("a plan needs at least one step".into()); } // TODO: do the uppercasing ONCE here, at plan time, and store it. // Build upper_steps from raw.steps so the per-record loop never redoes it. let upper_steps: Vec<String> = todo!(); Ok(CompiledPlan { upper_steps }) } } // Per-record runtime path — runs MANY times. It must only READ the frozen work. fn run(plan: &plan::CompiledPlan, records: &[u32]) { for r in records { // Reads the pre-computed names; does NOT uppercase here. println!("record {r}: {:?}", plan.upper_steps()); } } fn main() { let raw = plan::RawPlan { steps: vec!["read".into(), "filter".into(), "write".into()] }; match plan::compile(raw) { Ok(compiled) => run(&compiled, &[1, 2, 3]), Err(e) => println!("compile error: {e}"), } }

💡 Hint 1

You have raw.steps: Vec<String>. Map each step to its uppercase form and collect: raw.steps.iter().map(|s| s.to_uppercase()).collect(). The point is where it runs: once in compile, not once per record in run.

Show solution

let upper_steps: Vec<String> =
    raw.steps.iter().map(|s| s.to_uppercase()).collect();

This is the static/run-time split in one line: the uppercasing depends only on the plan, so it happens once in compile and is frozen into the handle. The per-record run loop only reads upper_steps. The real engine does exactly this at scale: it typechecks every CXL expression once at compile time and freezes each Arc<TypedProgram> into the plan, so the runtime pulls the typed program instead of re-typechecking on every record.

Faded: enforce the boundary from scratch

Your turn with much less scaffolding. Write the whole boundary: a RawPlan, a CompiledPlan with a private field, an accessor, the single compile constructor that rejects an empty plan, and a run that accepts only &CompiledPlan. The test of success: run(&raw) must be a compile error, and the only path to a CompiledPlan must be through compile.

rust // editable

reset ▶ run

mod plan {
  // TODO: a RawPlan with a public `steps: Vec<String>`.

  // TODO: a CompiledPlan with a PRIVATE field (so it can't be built outside).
  // Give it an accessor that hands out a shared reference.

  // TODO: the ONLY constructor. Reject an empty plan with Err; otherwise
  // freeze the validated data into the handle and return Ok.
  pub fn compile(raw: RawPlan) -> Result<CompiledPlan, String> {
      todo!()
  }
}

// TODO: run() must take &plan::CompiledPlan and nothing else.

fn main() {
  let raw = plan::RawPlan { steps: vec!["read".into(), "write".into()] };
  match plan::compile(raw) {
      Ok(compiled) => run(&compiled),
      Err(e) => println!("compile error: {e}"),
  }
  // Once you've written it, try uncommenting and watch it refuse to build:
  // run(&plan::RawPlan { steps: vec!["x".into()] });
}

mod plan { // TODO: a RawPlan with a public `steps: Vec<String>`. // TODO: a CompiledPlan with a PRIVATE field (so it can't be built outside). // Give it an accessor that hands out a shared reference. // TODO: the ONLY constructor. Reject an empty plan with Err; otherwise // freeze the validated data into the handle and return Ok. pub fn compile(raw: RawPlan) -> Result<CompiledPlan, String> { todo!() } } // TODO: run() must take &plan::CompiledPlan and nothing else. fn main() { let raw = plan::RawPlan { steps: vec!["read".into(), "write".into()] }; match plan::compile(raw) { Ok(compiled) => run(&compiled), Err(e) => println!("compile error: {e}"), } // Once you've written it, try uncommenting and watch it refuse to build: // run(&plan::RawPlan { steps: vec!["x".into()] }); }

Show solution

mod plan {
    pub struct RawPlan { pub steps: Vec<String> }

    pub struct CompiledPlan {
        steps: Vec<String>, // private — no outside construction
    }

    impl CompiledPlan {
        pub fn steps(&self) -> &[String] { &self.steps }
    }

    pub fn compile(raw: RawPlan) -> Result<CompiledPlan, String> {
        if raw.steps.is_empty() {
            return Err("a plan needs at least one step".into());
        }
        Ok(CompiledPlan { steps: raw.steps })
    }
}

fn run(plan: &plan::CompiledPlan) {
    println!("running {:?}", plan.steps());
}

The private steps field is the whole enforcement: outside the plan module, no one can write CompiledPlan { steps: ... }, so compile is the only door in. Demand &CompiledPlan at the runner and “run an unvalidated plan” becomes unrepresentable: exactly the move ValidatedPath made for screened paths two lessons back, now made for whole plans.

Solid on the plan/runtime boundary? Skip ahead →

// quick check

Why does PipelineExecutor accept &CompiledPlan rather than &PipelineConfig?

• CompiledPlan is smaller and faster to pass
• Because the only way to obtain a CompiledPlan is to run compile(), which does all validation and lowering, so requiring the handle makes 'run an uncompiled plan' a compile error
• Because PipelineConfig cannot be sent between threads

Retrieval checkpoint

✓ Checkpoint — the plan/runtime boundary

💡 Hint 1

Read the compiled.rs module doc first; it names CompiledPlan a “typed-handle output” and lists what it wraps. Then look at the executor’s compile_fail doctest to see the raw config being rejected on purpose.

What the tour establishes

CompiledPlan has private fields and a pub(crate) new, so only PipelineConfig::compile builds it. compile validates, lowers the DAG, and typechecks every CXL expression, freezing the results into the plan. The executor’s run_plan_with_readers_writers takes &CompiledPlan; its compile_fail doctest proves a &PipelineConfig is rejected. The per-record path then reads those frozen programs instead of recomputing them: decide once, run many.

// quick check

Which of these is plan-time (done once), not run-time (per record)?

• Evaluating a filter expression against the current record
• Typechecking each CXL expression and lowering the DAG
• Writing a record to the output file

// quick check

You have a PipelineConfig and pass it where run_plan_with_readers_writers expects &CompiledPlan. What happens, and why?

• It runs, re-validating the config at run time to be safe
• It fails to compile: a PipelineConfig is not a CompiledPlan, and you cannot forge one because its fields are private and compile() is the only constructor
• It compiles, then panics on the first record when the missing validation is noticed

// quick check

MISCONCEPTION check: which statement about the plan/runtime boundary is wrong and needs correcting?

• Anything that depends only on the plan (typecheck, lowering, schema binding) is settled once at compile time and frozen into CompiledPlan
• The executor re-validates the plan on each entry, so the typed handle is really just a performance convenience you could skip
• The per-record path reads the frozen typed programs (each transform's Arc<TypedProgram>) instead of recomputing them

// quick check

Spaced recall from the YAML-to-a-plan lesson: your YAML has a transform whose CXL reads a column no upstream node produces. When does the engine catch it, and what is the runtime handed?

• At run time, once a record arrives and the column comes up missing; the runtime parses the YAML itself
• At plan time: the validate stage rejects it before any record moves, and the runtime is only ever handed a CompiledPlan, never raw YAML/config
• Never: the unknown column is silently dropped from the output

You should be able to:

• You can list what CompiledPlan freezes in (lowered DAG, typechecked CXL, bound schemas, source hash)
• You can give two examples of plan-time work and two of run-time work, and explain why the split exists
• You can explain how requiring &CompiledPlan at the executor makes an uncompiled run a compile error
• You can predict the compile error from handing the runner a raw plan, and say why the type is the proof

Verify in the checkout:

grep -n 'pub struct CompiledPlan' crates/clinker-plan/src/plan/compiled.rs
grep -n 'pub fn compile' crates/clinker-plan/src/config/pipeline.rs
grep -n 'compile_fail' crates/clinker-exec/src/executor/mod.rs
grep -n 'pub fn run_plan_with_readers_writers' crates/clinker-exec/src/executor/mod.rs

Connections

You can now name what CompiledPlan freezes in, sort plan-time work from run-time work, explain why the executor accepts only the typed handle, and predict the compile error that makes “run an unvalidated plan” unrepresentable. You’ve seen the engine concentrate all its “is this valid?” work behind one typed handle. But plenty can still go wrong at run time: a bad cast, a file error, an internal invariant tripped. How the engine classifies those failures, and which ones abort versus get quarantined, is the next lesson.

Prerequisites

• Span-preserving parse & strictness

→

Leads to

• Error handling

Go deeper on the Rust (optional; compile returns a Result, and the ?-sequenced fallible stages behind the boundary are the same mechanics taught from first principles in The Rust Book):

• Want Result and the ? operator drilled on a standalone parser? Read the docs →

Glossary terms used: plan, runtime / executor, compile-time validation, Result.

Mark this lesson complete