Skip to content
Clinker Academy

Trace one record end-to-end

You have all the pieces now: Value, Record, Schema, the CompiledPlan, the DAG, the dispatch. This lesson threads them together by following one customer, Alice, from a line in a CSV file to a line in the output, naming every part of the engine she touches. You won’t read engine internals this time. You’ll run the real trace and read what it shows.

What you’ll be able to do

Section titled “What you’ll be able to do”
  • Trace one record (Alice) through all four customer_etl stages, naming the engine part each stage is.
  • Read the --dry-run output and the --explain plan, and point each printed line back to the stage that produced it.
  • Predict a record’s is_active and tier from the two CXL rules, then verify the prediction against the real run.
  • Explain why a string cell becomes a number in final_flag and not in the source: coercion on demand.

New terms in this lesson (each is also expanded inline at the point you first need it):

  • record: one row flowing through the engine.
  • Value: one cell of a record.
  • schema: the named columns a record is bound to.
  • CXL: the small expression language each transform runs.
  • node: one step in the pipeline graph.

Predict first

Section titled “Predict first”

Alice’s row

Section titled “Alice’s row”
clinker ·customer_etl.yaml example @19acdcb4

In the source CSV she’s one line:

customer_id,first_name,last_name,email,status,lifetime_value,zip_code
1001,Alice,Chen,alice.chen@acme.com,active,15200,94103

Stage 1: the source reads her

Section titled “Stage 1: the source reads her”

The source node reads the CSV and produces a Record: a Vec<Value> bound to the schema declared in the YAML. Every field is a string at this point, a Value::String:

clinker-record ·mod.rs ·Record type @19acdcb4 
schema:  [customer_id, first_name, last_name, email, status, lifetime_value, zip_code]
values:  [ "1001",     "Alice",    "Chen",   ...,    "active",  "15200",      "94103" ]

Stage 2: active_only flags her

Section titled “Stage 2: active_only flags her”

The first transform runs this CXL over every record:

emit is_active = status == "active"

Alice’s status is "active", so the comparison is true. The transform emits a new field (is_active = Bool(true)) and passes the enlarged record downstream. Her row now carries an eighth value.

Stage 3: final_flag tiers her

Section titled “Stage 3: final_flag tiers her”

The second transform:

emit tier = if lifetime_value.to_int() > $vars.gold_threshold then "gold" else "standard"

Here lifetime_value ("15200") is finally turned into a number by .to_int(), then compared against $vars.gold_threshold (default 10000). 15200 > 10000 is true, so tier = "gold". This is the coercion we promised back in what is a record?: the string becomes an integer exactly when a transform needs it to, not before.

Stage 4: the output writes her

Section titled “Stage 4: the output writes her”

The output node writes the record, now with is_active and tier added, to ./output/customers.csv. Alice leaves the engine as:

... ,active,15200,94103,true,gold

That’s nine cells: the original seven plus is_active and tier, exactly the count you predicted.

Use → Modify → Create

Section titled “Use → Modify → Create”

You’ve read the trace narrated. Now run it, change it, and predict a fresh one. Each rung adds exactly one new demand: read what’s there → change one input and predict → predict a trace you’ve never seen, then verify.

Use: run the trace as given

Section titled “Use: run the trace as given”

--dry-run runs the real engine and writes records to your terminal instead of a file; -n 5 caps it at the first five records:

Terminal window
cd examples/pipelines
cargo run -p clinker -- run customer_etl.yaml --dry-run -n 5

You saw the summary in Orientation: 5 total, 5 ok, 5 written, 0 dlq. Alice is one of those five, both ok and written. (Carol, who is inactive, still flows through; active_only only flags her is_active = false. Nothing is dropped here; filtering is a later topic.)

Pair it with --explain to see the plan the trace walks: the four nodes you named in the last lesson, in order, with no data read:

Terminal window
cargo run -p clinker -- run customer_etl.yaml --explain

Read the two together: --explain shows you the path (source → active_only → final_flag → output, DAG nodes: 4); --dry-run shows you a record that walked it. The nine cells in the dry-run row are the four stages above, made real.

Modify: change one input and predict the new trace

Section titled “Modify: change one input and predict the new trace”

Open customer_etl.yaml and find the variable the tiering rule reads:

vars:
  gold_threshold: { type: int, default: 10000 }

Change the default to 20000, save, and predict before you run: with the cutoff now 20000, does Alice’s 15200 still clear it? What does her tier become? Then run the same dry-run and check the last cell of her row.

💡 Hint 1

The rule is lifetime_value.to_int() > gold_threshold. Alice’s value is 15200. Compare 15200 > 20000. Only tier can change; is_active reads status, which you didn’t touch.

Show solution

15200 > 20000 is false, so Alice now leaves as tier = "standard". Only the last cell flipped; her other eight cells are identical, because you changed one variable that only final_flag reads. Restore default: 10000 when you’re done so the rest of the lesson lines up.

Create: predict a brand-new record’s trace, then verify

Section titled “Create: predict a brand-new record’s trace, then verify”

Here is a customer who is not in the file. Trace her on paper through all four stages, the same way you traced Alice, then add her line to ./data/customers.csv, run the dry-run, and confirm every cell.

2007,Dana,Okoro,dana@initech.com,inactive,42000,60601

Write down, before running: her is_active, her tier, and the full output row (with gold_threshold back at 10000).

💡 Hint 1

Two independent rules. is_active reads status: is Dana "active"? tier reads lifetime_value.to_int() vs 10000, regardless of is_active; the two transforms don’t talk to each other. An inactive customer still flows through and still gets a tier.

Show solution

Dana’s status is "inactive", so is_active = false. Her lifetime_value is 42000, and 42000 > 10000 is true, so tier = "gold". Being inactive does not suppress the tier; the rules are independent. She leaves as:

2007,Dana,Okoro,dana@initech.com,inactive,42000,60601,false,gold

Nine cells, exactly as the count predicts. Verify by appending her line to ./data/customers.csv and rerunning the dry-run (drop the -n 5 cap, or raise it, so her row is reached). The trap to avoid: assuming is_active = false means “dropped” or “no tier.” Nothing here filters, so she is written like everyone else.

Retrieval checkpoint

Section titled “Retrieval checkpoint”
💡 Hint 1

Apply the same two CXL rules. Is Bob’s status "active"? Is his lifetime_value (as an integer) greater than the gold_threshold of 10000?

Show solution

Bob’s status is "active", so is_active = true. His lifetime_value is 8400, and 8400 > 10000 is false, so tier = "standard". He leaves as ...,active,8400,10001,true,standard.

That’s A Record’s Journey

Section titled “That’s A Record’s Journey”

You can now follow a record from source to output and name every stage: a Record of Values bound to a Schema, produced by a source, transformed node-by-node as the executor walks the CompiledPlan’s DAG, and written by an output. That mental map is the spine of everything ahead.

Data & Representation revisits the first stage in depth: what a Value really costs, how records borrow instead of copy, and the ownership and lifetime rules that make the engine fast. Same journey, deeper each pass.

Go deeper on the Rust (optional, one-directional; the cell type Value is a closed enum, and The Rust Book builds enums like it from first principles, no engine framing):

Glossary terms used: record, Value, schema, CXL, node.