What a pipeline is

The question for this lesson: you have a CSV of customers, and you want Clinker to flag the active ones and tag each as “gold” or “standard”. Before you write anything, what is a pipeline, and how do you read one?

What you’ll be able to do

Describe the four-part mental model of a pipeline and use it to predict what a pipeline does.
Name the three node types in a basic pipeline and what each one is for.
Run a real pipeline and read its output.

🌱 New here? — YAML

YAML is a plain-text format for structured data. Indentation (spaces, never tabs) shows what belongs to what. A line like name: customers is a key and a value; a line starting with - is a list item. That’s nearly everything you need here.

The mental model

Every Clinker pipeline does the same three things, in order:

A source reads your data and turns each row into a record.
One or more nodes change those records. A transform computes new fields, a route splits them, an aggregate groups them.
An output writes the records back out to a file.

That’s the whole notional machine:

sources produce records → nodes transform/route/aggregate them → outputs write them. Anything that goes wrong with a row can be sent to a dead-letter queue instead of stopping the job.

You can predict what almost any pipeline does by tracing one record through that chain. You never need to know how Clinker does it inside, only what each node does.

Run a real one

Here is a complete, working pipeline. It reads customers, keeps a marker for the active ones, tags each by lifetime value, and writes the result.

pipeline.yaml // editable

pipeline:
  name: customer_etl
nodes:
  - type: source
    name: customers
    config:
      name: customers
      type: csv
      path: ./data/customers.csv
      options: { has_header: true }
      schema:
        - { name: status, type: string }
        - { name: lifetime_value, type: string }

  - type: transform
    name: active_only
    input: customers
    config:
      cxl: |
        emit is_active = status == "active"

  - type: transform
    name: final_flag
    input: active_only
    config:
      cxl: |
        emit tier = if lifetime_value.to_int() > 10000 then "gold" else "standard"

  - type: output
    name: results
    input: final_flag
    config:
      name: results
      type: csv
      path: ./output/customers.csv

> output appears here — predict, then run

Trace one record all the way through and you can predict the file before you open it. Later you’ll learn to validate a pipeline before running it, so you can check its structure without processing any data.

Reading the pieces

type: source … type: csv: read a CSV. schema: declares the columns and their types. Clinker does not guess types from the file; you state them.
type: transform … cxl:: run a small expression over every record. emit is_active = … adds a new column. (That little language is CXL, the focus of the next lesson.)
type: output: write the records to a file. Each node names the node it reads from with input:, which is how the chain is wired.

// quick check

In the mental model, what is a 'record'?

Try it

Change the final_flag transform so the gold threshold is 5000 instead of 10000. Predict which customers become “gold”, then reveal the expected result and check.

✓ Checkpoint

// quick check

Which node decides the value of the `tier` column?

Common mistakes

“The source figures out column types for me.” No: you declare each column’s type in the schema, and Clinker never infers them (you’ll see this up close in lesson 01).
“A record is the whole file.” A record is one row; a file is many records, and they flow through the nodes one at a time.
“The output node decides the new columns.” New columns come from the transform nodes; the output only writes whatever records reach it.

You should be able to:

Say the four-part mental model of a pipeline in one sentence
Name the three node types here and what each does
Point to the line that wires one node to the next

Verify in the checkout:

clinker run customer_etl.yaml --dry-run
clinker run customer_etl.yaml --explain