Skip to content
Clinker Academy

Through the DAG

--explain told you customer_etl is a graph of four nodes: DAG nodes: 4. Last lesson you watched the YAML become that graph: a CompiledPlan whose dag field holds the wired-up nodes. The plan describes the graph; the executor walks it. This lesson is how a record actually travels from one node to the next, and how the engine picks the right code to run at each stop. Shallow now; Execution & Memory is the deep version.

What you’ll be able to do

Section titled “What you’ll be able to do”
  • Draw customer_etl’s four nodes and name the order a record visits them.
  • Explain how the executor selects each node’s behavior: a single match over the closed PlanNode enum.
  • Predict the output of dispatching one node, and what the compiler does if a node kind is left unhandled.
  • Trace one record’s path through a small DAG in code, building the new field each node adds.

New terms in this lesson (each is also expanded inline at the point you first need it):

  • node: one step in the pipeline graph; a type: in your nodes: list.
  • DAG: a directed acyclic graph: nodes wired by edges, with no path that loops back on itself. The plan’s nodes form one, so the executor can walk them in a single forward pass.
  • streaming: records flow one at a time from node to node; the engine never collects the whole dataset in memory before the next node sees a row. This is the Mode: Streaming --explain reported.
  • tier: here, the customer-tier field final_flag adds to each record (e.g. gold/silver). Used as a concrete example of “a node enriches the record as it passes.”
  • backpressure: when a downstream node is slower than its upstream, a streaming engine lets the slow node set the pace rather than piling up unread records. Named here only so the word isn’t new in Execution & Memory.

Predict first

Section titled “Predict first”

The graph is a chain of nodes

Section titled “The graph is a chain of nodes”

For customer_etl, the node graph is a straight line; each node feeds the next. This is a DAG: nodes wired by edges, with no path that loops back on itself, so the executor can walk it in one forward pass.

 source            transform           transform          output
customers  ──▶  active_only   ──▶   final_flag   ──▶   results
 (read CSV)     (add is_active)     (add tier)        (write CSV)

Every node is one variant of a single closed enum, PlanNode: source, transform, output, and a handful of others (aggregate, route, combine…):

clinker-plan ·mod.rs ·PlanNode type @19acdcb4 
pub enum PlanNode {
    Source { /* ... */ },
    Transform { /* ... */ },
    Route { /* ... */ },
    Merge { /* ... */ },
    Sort { /* ... */ },
    Aggregation { /* ... */ },
    Output { /* ... */ },
    // ... one variant per kind of node a pipeline can contain
}

One match, every node kind

Section titled “One match, every node kind”

The executor walks the nodes and, for each, has to run the right logic: read for a source, evaluate CXL for a transform, write for an output. It does that with a single big match over the PlanNode enum:

clinker-exec ·dispatch.rs ·dispatch_plan_node fn @19acdcb4 
// the shape of it (real arms call into per-operator modules)
match node {
    PlanNode::Source { .. }    => /* read records   */,
    PlanNode::Transform { .. } => /* run the CXL     */,
    PlanNode::Output { .. }    => /* write records   */,
    // ... one arm per node kind
}

Because PlanNode is a closed set known at compile time, this match must handle every kind; the compiler refuses to build if a node type is left unhandled. There’s no plugin registry and no runtime lookup of “which handler?”; the set of operations is fixed and exhaustive. Why Clinker chose a closed enum over open, pluggable operators, and what that trades away, is one of the central decisions you’ll examine in Execution & Memory.

How a record moves

Section titled “How a record moves”

Records don’t teleport to the end; each node hands its outputs to the next. The source produces records and passes them downstream; active_only receives each one, runs its CXL, and passes the (now slightly larger) record on; final_flag does the same, adding the customer tier; the output writes whatever reaches it. One record at a time, in a streaming flow, which is exactly why --explain reported Mode: Streaming.

Streaming means the engine never buffers the whole dataset between nodes. If a downstream node is slower than its upstream, the slow node sets the pace. That pacing is called backpressure, and you’ll meet the real machinery for it in Execution & Memory. Here, just hold the picture: a record enters at the source and leaves at the output, growing as each transform enriches it.

Trace a record through the DAG

Section titled “Trace a record through the DAG”

You’ll understand the walk best by building a small version of it. Each rung below is a self-contained, runnable Rust program. We stage one new idea at a time: first dispatching one node, then completing the transform arm, then walking a whole chain.

Worked: dispatch one node

Section titled “Worked: dispatch one node”

Here is the executor’s core move in miniature: a closed PlanNode enum and a match that runs the right logic for each kind. Run it and watch each node kind take its own arm.

rust // editable

The takeaway: the executor never asks “which handler?” at runtime. The node’s kind is the answer, and match turns that kind into the right code path, checked exhaustively at compile time.

Completion: write the transform arm

Section titled “Completion: write the transform arm”

Below, the Source and Output arms are written for you. The Transform arm is the one to complete: a transform receives a record, runs its logic (here, adding a tier field), and returns the enriched record so the next node sees it. Fill it in.

rust // editable
💡 Hint 1
You have adds_tier (an &'static str) and rec. Return a Record with the same id but tier: Some(adds_tier). Struct update syntax, Record { tier: Some(adds_tier), ..rec }, copies the rest of the fields.
Show solution 
PlanNode::Transform { adds_tier } => {
    Record { tier: Some(adds_tier), ..rec }
}

A transform’s whole job in one line: take the record in, hand the enriched record out. The executor calls this once per record, and the result flows straight to the next node. That’s the streaming hand-off, with no dataset buffered in between.

Faded: walk the whole chain

Section titled “Faded: walk the whole chain”

Your turn with the stages wired together. Write run_dag, which walks a slice of nodes in order, threading one record through all of them (source, then each transform, then output) and returns the final record. Each node is dispatched with the dispatch you just saw; you write the loop that walks the DAG.

rust // editable
Show solution 
fn run_dag(nodes: &[PlanNode], start: Record) -> Record {
    let mut rec = start;
    for node in nodes {
        rec = dispatch(node, rec); // each node's output is the next node's input
    }
    rec
}

That’s the executor’s spine for a straight-line DAG: one record, walked forward node by node, each node’s output becoming the next one’s input. The real executor does far more. It streams many records, handles branching graphs, and dispatches into per-operator modules. But the shape, walk the nodes and pass the record along, is exactly this.

What the source actually does

Section titled “What the source actually does”

The miniature mirrors two real symbols. First, the node kinds are a closed enum, PlanNode; the dag field of the CompiledPlan you built last lesson holds these, wired into a graph:

clinker-plan ·mod.rs ·PlanNode type @19acdcb4 
pub enum PlanNode {
    Source { /* ... */ },
    Transform { /* ... */ },
    Output { /* ... */ },
    // ... one variant per kind of node a pipeline can contain
}

And the executor dispatches each with a single match, just like your dispatch:

clinker-exec ·dispatch.rs ·dispatch_plan_node fn @19acdcb4 
// real arms call into per-operator modules instead of inline code
match node {
    PlanNode::Source { .. }    => /* read records */,
    PlanNode::Transform { .. } => /* run the CXL  */,
    PlanNode::Output { .. }    => /* write records */,
    // ... one arm per node kind
}

What the compiler enforces: because PlanNode is closed, the match must cover every variant. Add a new node kind to the enum and forget an arm, and the build fails with a non-exhaustive-match error. You cannot ship an executor that silently ignores a node type.

What a junior might misread: thinking each node “knows how to run itself” via a stored function, or that the engine looks a handler up by name. It doesn’t. The node carries only its configuration; the executor’s match supplies the behavior. The set of behaviors is fixed at compile time, which is what makes the exhaustiveness check possible.

Why-bridge: closed dispatch, and what it buys

Section titled “Why-bridge: closed dispatch, and what it buys”

Why does the engine model node kinds as one closed enum dispatched by a match, instead of an open registry of pluggable operators? Because a closed set is one the compiler can reason about completely: every node kind is known at build time, so the exhaustive match is a guarantee that no kind is silently unhandled. The cost is extensibility. You can’t drop in a new operator without editing the enum and recompiling.

That trade is the opposite of a plugin architecture, and it’s deliberate: Clinker buys compile-time exhaustiveness and predictable dispatch at the price of runtime pluggability.

Retrieval checkpoint

Section titled “Retrieval checkpoint”

Connections

Section titled “Connections”

You’ve seen the parts: records, the plan, the graph, the dispatch. You can draw customer_etl’s four nodes, explain how one match over the closed PlanNode enum selects each node’s behavior, and walk a record through a small DAG in code. Time to put one real record through all of them at once.

Go deeper on the Rust (optional, one-directional; the same enum/match mechanics taught from first principles in The Rust Book):

Glossary terms used: node, record.