Capstone — build a pipeline end to end

You’ve learned every piece. This capstone puts them together into one real pipeline: no new features, just the whole track working as a unit. You’ll build it, validate it before it runs, and run it.

The brief

Build a customer pipeline that:

1. Reads customers.csv with a declared schema (status, lifetime_value).
2. Keeps only active customers and computes a tier: "gold" when lifetime_value is over a threshold, else "standard". The threshold must be a pipeline variable so a channel can change it per tenant.
3. Routes gold customers down one branch and everyone else down another.
4. Writes two outputs: gold.csv and standard.csv.
5. Summarizes per tier (a count of customers and their total lifetime value) with an aggregate, written to summary.csv.
6. Continues past bad rows, sending failures to a dead-letter queue.

Each requirement is something the track already taught:

Requirement	From lesson
Source + declared schema	01: Sources & schema
filter + emit + the if/then/else tier	02: Transforms & CXL
route with a condition and a default	03: Composition
Per-tier summary with an aggregate (count + sum)	05: Groups & windows
Threshold as a $vars value a channel can override	04: Channels
error_handling + dlq	07: Errors & the DLQ
Confirm it with --dry-run / --explain	08: Validate before you run

Build it

Start from this scaffold (the source is given) and add the transform, the route, the aggregate, the three output nodes, and the error_handling block. Then check yourself against the reference below.

pipeline:
  name: customer_capstone
  vars:
    gold_threshold: { type: int, default: 10000 }
nodes:
  - type: source
    name: customers
    config:
      name: customers
      type: csv
      path: ./data/customers.csv
      options: { has_header: true }
      schema:
        - { name: status, type: string }
        - { name: lifetime_value, type: string }
  # TODO: classify (filter active; emit tier from $vars.gold_threshold)
  # TODO: route by_tier (gold vs default standard)
  # TODO: aggregate tier_summary (group_by tier; count + sum), also reading from classify
  # TODO: two outputs — gold.csv and standard.csv
  # TODO: summary output — summary.csv from tier_summary
# TODO: error_handling with a dlq

Run the reference

Here is the complete pipeline. Predict what lands in each output, then run it.

✋ Predict first

The three customers are (active, 80000), (inactive, 1200), (active, 4000) and gold_threshold is 10000. After filter-active and tier, what's in gold.csv and standard.csv, what does the per-tier summary.csv count and total, and what happened to the inactive customer?

Reveal the expected result

gold.csv:
status,lifetime_value,tier
active,80000,gold

standard.csv:
status,lifetime_value,tier
active,4000,standard

summary.csv:
tier,customers,total_value
gold,1,80000
standard,1,4000

pipeline.yaml // editable

reset ⌖ explain ✓ dry-run ▶ run

pipeline: name: customer_capstone vars: gold_threshold: { type: int, default: 10000 } # channel-overridable (lesson 04) nodes: - type: source name: customers config: name: customers type: csv path: ./data/customers.csv options: { has_header: true } schema: - { name: status, type: string } - { name: lifetime_value, type: string } - type: transform name: classify input: customers config: cxl: | filter status == "active" emit status = status emit lifetime_value = lifetime_value emit tier = if lifetime_value.to_int() > $vars.gold_threshold then "gold" else "standard" - type: route name: by_tier input: classify config: mode: exclusive conditions: gold: 'tier == "gold"' default: standard - type: aggregate name: tier_summary input: classify config: group_by: [tier] cxl: | emit tier = tier emit customers = count(*) emit total_value = sum(lifetime_value.to_int()) - type: output name: gold_customers input: by_tier.gold config: name: gold_customers type: csv path: ./output/gold.csv - type: output name: standard_customers input: by_tier.standard config: name: standard_customers type: csv path: ./output/standard.csv - type: output name: summary input: tier_summary config: name: summary type: csv path: ./output/summary.csv error_handling: strategy: continue dlq: path: ./output/errors.csv include_reason: true

The inactive customer was dropped by filter status == "active", so it appears in neither output. Of the two active customers, 80000 clears the 10000 threshold (gold) and 4000 does not (standard), and each is routed to its own file. The same classified rows also fan out to the aggregate, which groups by tier and reports one customer and the matching total in each group, so summary.csv counts the two tiers without the dropped inactive row ever appearing.

✋ Predict first

Without editing anything else, how would you make the 4000 customer count as gold too, using something from lesson 04, not by editing the transform?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

Apply a channel that overrides gold_threshold to something below 4000 (say 3000). Because the transform reads $vars.gold_threshold rather than a hardcoded number, the threshold is a per-deployment knob, with no change to the pipeline’s logic. That’s the channel pattern from lesson 04, now paying off in a complete pipeline.

Validate before you ship it

The habit from lesson 08: never ship a pipeline you haven’t validated. Before a real run,

clinker run customer_capstone.yaml --dry-run     # config + schema valid?
clinker run customer_capstone.yaml --explain     # the plan: nodes, wiring, outputs

--dry-run catches a misspelled input:, an undeclared schema type, or a CXL compile error in a second; --explain shows the shape (source → classify → route + aggregate → three outputs) so you can confirm the wiring matches the brief, including that classify fans out to both the route and the aggregate.

Checkpoint: the whole track

✓ Checkpoint

// quick check

In the capstone, why is the tier threshold written as `$vars.gold_threshold` instead of the literal 10000?

• So a channel can override it per tenant without editing the pipeline's logic
• Because CXL can't compare against a literal number
• To make the pipeline run faster

// quick check

Both `by_tier` and `tier_summary` list `input: classify`. Why does the summary count only two customers, not three?

• Because `classify` filtered the inactive customer out first, so both the route and the aggregate read the same already-filtered rows
• Because an aggregate always skips the first row it sees
• Because routing to gold.csv removes rows before the aggregate runs

// quick check

You run the capstone and it exits with code 2. From lesson 09, did it succeed?

• Yes, it completed; exit 2 means some rows were dead-lettered, not that the run failed
• No, any non-zero exit code is a failure
• It means the configuration was invalid

Common mistakes

• Hardcoding the threshold. Use $vars.gold_threshold so a channel can tune it per tenant.
• Routing without a default. Give the route a default: so non-gold rows have somewhere to go.
• Shipping without --dry-run. Validate first; a typo’d input: or undeclared type is a one-second fix before the run, not after.

You should be able to:

• Declare a source schema (no type inference), lesson 01
• Filter and compute fields with CXL, including if/then/else, lesson 02
• Wire nodes and split records with a route + default, lesson 03
• Summarize per group with an aggregate (count + sum), lesson 05
• Make a value channel-overridable with $vars, lesson 04
• Continue past bad rows with error_handling + a dlq, lesson 07
• Validate with --dry-run / --explain before running, lesson 08

Verify in the checkout:

clinker run customer_capstone.yaml --dry-run
clinker run customer_capstone.yaml --explain
clinker run customer_capstone.yaml

Where to go next

You can now read, build, validate, and operate a real Clinker pipeline end to end. If you ever want to see how the engine does this underneath, how a route becomes branches in an execution plan, or how records flow without copying, the Engine and Rust tracks pick up exactly there. You never need them to build pipelines; they’re there when you’re curious.