The plan/runtime boundary

The executor is the engine’s hot, parallel, memory-bounded core — the part you least want running a half-checked plan. The last three lessons built up the validated pieces: a screened path, a strict located parse. This lesson is where they converge into a single typed handle — CompiledPlan — and where the engine makes “this plan is fully compiled” the same kind of unforgeable, compiler-checked fact that ValidatedPath made of “this path is screened.”

Prerequisites

• Span-preserving parse & strictness

→

Leads to

• Error handling

You’ll be able to: explain what CompiledPlan wraps, what the engine decides once at plan time versus per record at run time, and why the executor’s entry point refuses a raw config.

A handle that means “compiled”

CompiledPlan is the output of compilation — and like ValidatedPath, its fields are private, so the only way to hold one is to have gone through the compiler:

clinker-plan ·compiled.rs ·CompiledPlan type @47d2e12

/// CompiledPlan is the typed-handle output of PipelineConfig::compile.
#[derive(Debug)]
pub struct CompiledPlan {
    dag: ExecutionPlanDag,             // the lowered execution DAG
    config: PipelineConfig,            // the validated config
    artifacts: CompileArtifacts,       // per-node CXL typecheck results, bound schemas
    channel_identity: Option<ChannelIdentity>,
    pipeline_hash: [u8; 32],           // BLAKE3 of the source YAML
}

Everything expensive and checkable has already happened by the time this struct exists. The DAG is lowered (nodes resolved, topology fixed). Every CXL expression is typechecked, its typed program stored in artifacts. Schemas are bound. The source is hashed. The struct is read-only from the outside — accessors like dag(), config(), and artifacts() hand out shared references, nothing more.

The static/runtime split: decide once, run many

This is the architectural heart of the lesson. Work in a streaming engine falls into two buckets:

• Plan-time (static), done once per job: parse, validate, resolve references, lower the DAG, typecheck every CXL expression, bind schemas. Expensive, but paid a single time.
• Run-time (dynamic), done once per record: pull a record, evaluate the already-typechecked expressions, route it, write it. This runs millions of times.

CompiledPlan is the membrane between them. Anything that can be settled before the first record flows is settled at compile time and frozen into the handle, so the per-record path never repeats it. The doc on artifacts() says exactly this: “The runtime executor reads this map directly to pull each transform’s Arc<TypedProgram> instead of re-typechecking.” The executor never typechecks a CXL expression — that was done once, at compile time, and the result is in the handle. (Lesson 23 shows the same “compile once, run per record” idea inside CXL itself.)

The nodes the DAG is built from are themselves self-describing — each PlanNode owns its topology, its display form, and its execution payload, plus a span back to the source YAML for diagnostics:

clinker-plan ·mod.rs ·PlanNode type @47d2e12

pub enum PlanNode {
    Source { name: String, span: Span, /* ... */ },
    Transform { name: String, span: Span, /* ... */ },
    Output { name: String, span: Span, /* ... */ },
    // ... each variant owns its topology, display, and execution payload
}

The executor accepts the handle and nothing else

Here’s the enforcement. The executor’s public entry point takes &CompiledPlan. It does not accept a PipelineConfig. And a compile_fail doctest pins that refusal so it can never silently regress:

clinker-exec ·mod.rs ·run_plan_with_readers_writers fn @47d2e12

/// Accepts the typed `CompiledPlan` handle returned by `PipelineConfig::compile`.
///
/// ```compile_fail
///     let _ = PipelineExecutor::run_plan_with_readers_writers(
///         cfg, // ← should be &CompiledPlan, not &PipelineConfig
///         // ...
///     );
/// ```
pub fn run_plan_with_readers_writers<W: Into<WriterRegistry>>(
    plan: &CompiledPlan,
    // ...
) { /* ... */ }

Trace the guarantee, exactly as with ValidatedPath: to run, you need a &CompiledPlan; to get a CompiledPlan, you must call PipelineConfig::compile (the only constructor); and compile is where all validation lives. So “the executor ran an unvalidated plan” is not a bug you guard against at run time — it’s a program that does not compile. The type is the proof that compilation happened.

Build the boundary yourself

The whole pattern fits in one screen. A raw plan, a compile step that validates and bakes a result, a private-fielded handle, and a runner that accepts only the handle:

rust // editable

reset ▶ run

mod plan {
  pub struct RawPlan { pub steps: Vec<String> }

  // Typed handle — private fields, so only `compile` can build one.
  pub struct CompiledPlan {
      steps: Vec<String>,
      step_count: usize, // computed ONCE at compile time
  }

  impl CompiledPlan {
      pub fn step_count(&self) -> usize { self.step_count }
      pub fn steps(&self) -> &[String] { &self.steps }
  }

  // The ONLY constructor: validate + lower once, freeze into the handle.
  pub fn compile(raw: RawPlan) -> Result<CompiledPlan, String> {
      if raw.steps.is_empty() {
          return Err("a plan needs at least one step".into());
      }
      let step_count = raw.steps.len();
      Ok(CompiledPlan { steps: raw.steps, step_count })
  }
}

// The executor accepts ONLY the compiled handle.
fn run(plan: &plan::CompiledPlan) {
  println!("running {} steps: {:?}", plan.step_count(), plan.steps());
}

fn main() {
  let raw = plan::RawPlan { steps: vec!["read".into(), "filter".into(), "write".into()] };
  match plan::compile(raw) {
      Ok(compiled) => run(&compiled),
      Err(e) => println!("compile error: {e}"),
  }

  // Try this: skip compilation and hand the runner a raw plan.
  // let raw2 = plan::RawPlan { steps: vec!["read".into()] };
  // run(&raw2);
  // ^ won't compile: run() wants &CompiledPlan, and you can't forge one —
  //   its fields are private; only plan::compile produces one.
}

mod plan { pub struct RawPlan { pub steps: Vec<String> } // Typed handle — private fields, so only `compile` can build one. pub struct CompiledPlan { steps: Vec<String>, step_count: usize, // computed ONCE at compile time } impl CompiledPlan { pub fn step_count(&self) -> usize { self.step_count } pub fn steps(&self) -> &[String] { &self.steps } } // The ONLY constructor: validate + lower once, freeze into the handle. pub fn compile(raw: RawPlan) -> Result<CompiledPlan, String> { if raw.steps.is_empty() { return Err("a plan needs at least one step".into()); } let step_count = raw.steps.len(); Ok(CompiledPlan { steps: raw.steps, step_count }) } } // The executor accepts ONLY the compiled handle. fn run(plan: &plan::CompiledPlan) { println!("running {} steps: {:?}", plan.step_count(), plan.steps()); } fn main() { let raw = plan::RawPlan { steps: vec!["read".into(), "filter".into(), "write".into()] }; match plan::compile(raw) { Ok(compiled) => run(&compiled), Err(e) => println!("compile error: {e}"), } // Try this: skip compilation and hand the runner a raw plan. // let raw2 = plan::RawPlan { steps: vec!["read".into()] }; // run(&raw2); // ^ won't compile: run() wants &CompiledPlan, and you can't forge one — // its fields are private; only plan::compile produces one. }

Uncomment the last block and the build fails: run wants a &CompiledPlan, you have a RawPlan, and you cannot hand-build a CompiledPlan because its fields are private. The only way forward is through compile — which is the only place validation happens. That refusal is the plan/runtime boundary in one compiler error.

Solid on the plan/runtime boundary? Skip ahead →

// quick check

Why does PipelineExecutor accept &CompiledPlan rather than &PipelineConfig?

• CompiledPlan is smaller and faster to pass
• Because the only way to obtain a CompiledPlan is to run compile(), which does all validation and lowering — so requiring the handle makes 'run an uncompiled plan' a compile error
• Because PipelineConfig cannot be sent between threads

Inspect the boundary

✓ Checkpoint — the plan/runtime boundary

💡 Hint 1

Read the compiled.rs module doc first — it names CompiledPlan a “typed-handle output” and lists what it wraps. Then look at the executor’s compile_fail doctest to see the raw config being rejected on purpose.

What the tour establishes

CompiledPlan has private fields and a pub(crate) new, so only PipelineConfig::compile builds it. compile validates, lowers the DAG, and typechecks every CXL expression, storing the results in artifacts. The executor’s run_plan_with_readers_writers takes &CompiledPlan; its compile_fail doctest proves a &PipelineConfig is rejected. The per-record path then reads those artifacts instead of recomputing them — decide once, run many.

// quick check

Which of these is plan-time (done once), not run-time (per record)?

• Evaluating a filter expression against the current record
• Typechecking each CXL expression and lowering the DAG
• Writing a record to the output file

You should be able to:

• You can list what CompiledPlan freezes in (lowered DAG, typechecked CXL, bound schemas, source hash)
• You can give two examples of plan-time work and two of run-time work, and explain why the split exists
• You can explain how requiring &CompiledPlan at the executor makes an uncompiled run a compile error

Verify in the checkout:

grep -n 'pub struct CompiledPlan' crates/clinker-plan/src/plan/compiled.rs
grep -n 'pub fn compile' crates/clinker-plan/src/config/pipeline.rs
grep -n 'compile_fail' crates/clinker-exec/src/executor/mod.rs
grep -n 'pub fn run_plan_with_readers_writers' crates/clinker-exec/src/executor/mod.rs

You’ve seen the engine concentrate all its “is this valid?” work behind one typed handle. But plenty can still go wrong at run time — a bad cast, a file error, an internal invariant tripped. How the engine classifies those failures, and which ones abort versus get quarantined, is the next lesson.

Mark this lesson complete