Build & run Clinker

The fastest way to start understanding a system is to run it once, end to end, and watch it do something real. Before we open a single source file, let’s get Clinker compiling and move some actual data through it, then read what the engine tells us it did.

Prerequisites

none — start here

→

Leads to

• The compiler loop

What you’ll be able to do

• Build the engine from a clinker checkout with one cargo command.
• Run the example pipeline two ways: --explain (plan only, no data moves) and --dry-run (data moves, output goes to the terminal).
• Read an execution plan and name what each line reports: the DAG node count, the transform count, and the run mode.
• Read the --dry-run summary line and name what each of total / ok / written / dlq counts.
• Predict how the plan and the summary change when you alter one flag (-n) or one declared variable ($vars.gold_threshold), then verify against the real output.

New terms in this lesson (each is named here before it appears below):

• pipeline: a whole job, described in YAML.
• node, one step in that job (a source, a transform, an output).
• DAG: the directed, acyclic graph the nodes form.
• CXL, the small expression language a transform runs per record.
• plan: the compiled, validated form of the job, printed by --explain.
• dead-letter queue (DLQ), where rejected records go; the dlq count in the summary.
• dry-run: a real run whose output is redirected to your terminal.

What Clinker is (the one-paragraph version)

Clinker is a bounded-memory, single-process batch executor for finite ETL jobs. You describe a job as a pipeline in YAML: a set of nodes (a source, some transforms, an output) wired into a graph. Per-record logic is written in a small expression language called CXL. You hand the pipeline to the clinker command; it reads records from the source, pushes them through the graph, writes the output, and exits. Finite and batch: the sources end, the job drains, the process stops.

That’s the whole mental model for now. We’ll earn every word of it over the coming lessons.

The example we’ll run

Clinker ships runnable example pipelines. We’ll use the canonical one:

clinker ·customer_etl.yaml example @19acdcb4

It’s a customer ETL job: a CSV of customers in, a CSV of flagged customers out, with two transforms in between.

nodes:
  - type: source        # read customers.csv
    name: customers
    config: { type: csv, path: ./data/customers.csv, ... }

  - type: transform     # add an is_active flag
    name: active_only
    input: customers
    config:
      cxl: |
        emit is_active = status == "active"

  - type: transform     # classify into a gold/standard tier
    name: final_flag
    input: active_only
    config:
      cxl: |
        emit tier = if lifetime_value.to_int() > $vars.gold_threshold then "gold" else "standard"

  - type: output        # write the result
    name: results
    input: final_flag
    config: { type: csv, path: ./output/customers.csv }

Four nodes (source → transform → transform → output), a tiny pipeline that is nonetheless a complete Clinker job. The two transforms run CXL per record; $vars.gold_threshold is a declared variable (default 10000) the pipeline reads instead of hard-coding the cutoff. The input is small and human-readable:

customer_id,first_name,last_name,email,status,lifetime_value,zip_code
1001,Alice,Chen,alice.chen@acme.com,active,15200,94103
1002,Bob,Martinez,bob.m@globex.com,active,8400,10001
1003,Carol,Johnson,carol.j@example.com,inactive,3200,60601

Alice is active with a lifetime value above the gold_threshold (default 10000), so she’ll be flagged gold; Bob is active but below it, so standard; Carol is inactive.

Build it

Clinker pins its toolchain (a rust-toolchain.toml selects the exact Rust version), so rustup installs the right compiler automatically the first time. From your clinker checkout:

cargo build -p clinker

The first build compiles the whole workspace and takes a few minutes; after that, builds are incremental and fast. (The compiler loop is all about that fast inner loop.)

Predict first

Before you run anything, predict what the engine will say. You’ve read the four-node pipeline above; that’s enough to call the plan.

✋ Predict first

You're about to run `clinker run customer_etl.yaml --explain`, which compiles the pipeline into a plan and prints it WITHOUT moving any data. The pipeline has four nodes: one source, two transforms, one output. Before you read on: how many DAG nodes will the plan report? How many transforms? And does --explain process any records? Write down all three answers.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

The plan reports exactly what the YAML declared, counted by kind:

• DAG nodes: 4, one per node in the pipeline (source + two transforms + output).
• Transforms: 2: only the two type: transform nodes; the source and output are counted separately, not as transforms.
• Records processed: 0. --explain compiles and validates the job, then prints the plan and stops. No source is read, no output is written.

The trap is assuming --explain runs a small slice of the data to produce the plan. It doesn’t run any data; the plan is built purely from the pipeline’s structure. That’s the whole point of the flag: it proves the job is well-formed (the nodes wire up, the CXL type-checks) before a single record moves. You’re checking the shape of the job, not its output. Hold those three numbers; the run below earns them.

Run it

Two ways to run a pipeline. Start with the one that doesn’t touch any data.

1. See the plan, without executing (--explain). Run the example from the examples/pipelines/ directory (so the pipeline’s ./data/... paths resolve):

cd examples/pipelines
cargo run -p clinker -- run customer_etl.yaml --explain

Clinker compiles the pipeline into an execution plan and prints it, but runs nothing:

=== Execution Plan ===

Mode: Streaming
Transforms: 2
Output projections: 1
DAG nodes: 4
arbitration: BackPressurePreferred -> Priority

Source DAG:
  Tier 0: customers

Four DAG nodes, two transforms, “Streaming” mode: exactly the three numbers you predicted. You’re looking at the plan, the proof that the job is well-formed, before any record moves. We’ll come back to this view in Read a plan with —explain, and to why planning is separate from running much later.

2. Actually move data (--dry-run). A dry run processes records and writes the result to your terminal instead of to the output file:

cargo run -p clinker -- run customer_etl.yaml --dry-run -n 5

INFO clinker: Pipeline complete: 5 total, 5 ok, 5 written, 0 dlq

Five records in, five processed, five written, zero rejected, and the process exits 0. That summary line is Clinker telling you the finite job ran clean, counted four ways:

• total: records read from the source.
• ok: records that passed through every node without error.
• written: records that reached the output.
• dlq: records routed to the dead-letter queue (rejected, a bad row that couldn’t be processed). Zero here means nothing was rejected; we’ll meet the DLQ properly soon.

The -n 5 is a flag that caps the run at five records. Drop it and the whole source runs; raise or lower the number and the counts move with it. That’s exactly the knob the apply section below has you turn.

Apply: Use → Modify → Create

You’ve run the example as given (Use). Now change one thing at a time and predict the new output before you run it. That predict-then-check loop is how you learn to read the engine, not just operate it.

Modify 1: turn the -n knob

Predict, then run. You ran --dry-run -n 5 and saw 5 total, 5 ok, 5 written, 0 dlq. Now change the cap to -n 2:

cargo run -p clinker -- run customer_etl.yaml --dry-run -n 2

Predict first: what does the summary line read now?

✋ Predict first

You changed -n 5 to -n 2 on the same clean dataset. What does the summary line 'N total, N ok, N written, N dlq' read now?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the expected summary

INFO clinker: Pipeline complete: 2 total, 2 ok, 2 written, 0 dlq

-n caps how many records the source produces, so total drops to 2. The data is clean, so ok and written track total, and dlq stays 0. The shape of the line never changes; only the counts do. If your ok or written had come out below total, that would mean records were dropped or rejected along the way; on this clean example they don’t.

Modify 2: change a declared variable

The pipeline classifies a customer as gold when lifetime_value > $vars.gold_threshold. The threshold is a declared variable at the top of the YAML:

pipeline:
  vars:
    gold_threshold: { type: int, default: 10000 }

Bob’s lifetime value is 8400, so at the default he’s standard. Lower the declared default below Bob’s value (edit the file, change 10000 to 8000), then re-run the dry run:

cargo run -p clinker -- run customer_etl.yaml --dry-run -n 5

Predict first: does changing the threshold change any of the four summary counts? And does Bob’s tier change?

✋ Predict first

You lowered gold_threshold from 10000 to 8000. Bob's lifetime_value is 8400. Does the total/ok/written/dlq summary change? Does Bob's tier change from 'standard' to 'gold'?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the expected behavior

The summary counts do not change. The same records flow through; lowering the threshold reclassifies a value, it doesn’t add, drop, or reject any row. So you still see 5 total, 5 ok, 5 written, 0 dlq.

Bob’s tier does change to gold: 8400 > 8000 is now true, where 8400 > 10000 was false. The summary line counts records, not field values, so a reclassification is invisible there; you’d see it in the per-record output, not the counts. That’s the distinction the misconception quiz below tests: the summary tells you the job ran clean, not what each field became.

Create: a predict-then-verify of your own

Now reason about the plan with no worked answer in front of you. Before running anything, write down what --explain would report for DAG nodes and Transforms if you imagine adding one more type: transform node to the pipeline (you don’t have to edit the file, just predict the two numbers). Then re-run

cargo run -p clinker -- run customer_etl.yaml --explain

on the unmodified four-node pipeline and check your reasoning against the real plan (DAG nodes: 4, Transforms: 2): your predicted five-node numbers should be exactly one higher in each line.

✋ Predict first

If the pipeline had FIVE nodes (one source, THREE transforms, one output), what would --explain report for 'DAG nodes' and for 'Transforms'?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

DAG nodes: 5 (one per node, every kind counted) and Transforms: 3 (only the type: transform nodes). The two counts answer different questions: DAG nodes is the total step count; Transforms is the subset that runs CXL. Source and output are nodes but not transforms, which is why the real four-node plan reads DAG nodes: 4 but Transforms: 2. If you predicted Transforms: 5, you conflated “every node” with “every transform”; the plan keeps them as separate lines on purpose.

Retrieval checkpoint

✓ Checkpoint

// quick check

What does it mean that Clinker is a *finite batch* executor?

• It runs forever as a service, processing data as it arrives
• The sources end, the job drains, and the process exits
• It can only process exactly one record at a time

// quick check

You run the example with `--explain`. How many records does it read from the source before printing the plan?

• All of them, because it needs the data to build the plan
• A small sample (the first few rows)
• None: `--explain` compiles and validates the job, then prints the plan without moving any data

// quick check

The plan prints `DAG nodes: 4` and `Transforms: 2` for the same pipeline. Which statement is the MISCONCEPTION to correct?

• `DAG nodes` counts every node; `Transforms` counts only the `type: transform` nodes, so the two numbers differ
• `DAG nodes` and `Transforms` should always be equal, so a 4-vs-2 plan means something is misconfigured
• The source and output are nodes in the DAG but are not counted as transforms

// quick check

You lower `gold_threshold` so a customer who was `standard` becomes `gold`. What happens to the `--dry-run` summary line `N total, N ok, N written, N dlq`?

• `written` goes up because a `gold` record is an extra output
• It is unchanged: the same records flow through; only a field VALUE was reclassified, and the summary counts records, not field values
• `dlq` goes up because reclassifying a record rejects the old version

You should be able to:

• Clinker built (`cargo build -p clinker` succeeds)
• `--explain` prints an execution plan with 4 DAG nodes and 2 transforms, and processes no records
• `--dry-run -n 5` reports `5 total, 5 ok, 5 written, 0 dlq` and exits 0; `-n 2` reports `2 total, ...`
• You can name what `total / ok / written / dlq` each count
• You predicted how the plan and the summary change when you turn the `-n` knob or override `gold_threshold`

Verify in the checkout:

cd examples/pipelines
cargo run -p clinker -- run customer_etl.yaml --explain
cargo run -p clinker -- run customer_etl.yaml --dry-run -n 5
cargo run -p clinker -- run customer_etl.yaml --dry-run -n 2

You just ran a four-node DAG end to end and learned to read both what the engine plans to do (--explain) and what it did (--dry-run). Each of those nodes (the source, the two CXL transforms, the output) is a door we’ll open in later lessons. Next: the fast edit-and-check loop you’ll live in while working on the engine.

From the pipeline author’s side (optional, one-directional; the same example job taught from the YAML side, no engine internals):

Want to start from the pipeline author's side, writing the YAML, not building the engine? Skip ahead →

Mark this lesson complete