Transforms & CXL — compute new fields

The question for this lesson: your source gives you the columns that were in the file. But you almost always need new columns: a flag, a category, a cleaned-up value. How do you compute a new field for every record?

What you’ll be able to do

• Read and write a transform node that adds fields with CXL.
• Use emit, comparisons, if/then/else, and a method like .to_int().
• Pull a value out of a channel with $vars, so one pipeline serves many deployments.
• Validate a pipeline’s config and declared types with --dry-run before you run it.

This lesson assumes you’ve met the pipeline mental model and the source/transform/output nodes from What a pipeline is. If “record” or “node” feels fuzzy, start there.

Predict first

✋ Predict first

A transform contains: emit is_active = status == 'active'. The source row has status = 'active'. What new column appears, and what is its value? What about a row with status = 'churned'?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

emit adds a column named is_active. The expression status == "active" compares the status field to the text "active" and produces a true/false value. So the "active" row gets is_active = true; the "churned" row gets is_active = false.

A transform runs its CXL once per record, top to bottom, and each emit adds (or overwrites) one column on that record. That’s the whole idea. The rest is which expressions you write.

What CXL is

CXL is the little language inside a transform’s cxl: block. You write one statement per line. The ones you’ll use constantly:

Statement	What it does	Example
emit <name> = <expr>	add or overwrite a column	emit total = price * qty
let <name> = <expr>	a temporary value (not a column)	let net = price - discount
filter <condition>	drop rows where the condition is false	filter status == "active"
distinct	keep only unique rows	distinct by id

Expressions use the operators you’d expect (+ - * /, == != > <, and or not), plus if <cond> then <a> else <b>, and methods on values like .to_int(), .upper(), .trim().

🌱 New here? — why .to_int()?

A value read from a CSV starts life as text. "4000" is the characters 4-0-0-0, not the number 4000, so "4000" > 10000 would compare text, not numbers. .to_int() converts the text to a whole number first, so the comparison means what you expect. (There’s also .to_float(), .to_string(), and lenient try_int() that yields nothing instead of an error on bad input.)

Use: a real transform

This is the canonical customer_etl pipeline. The second transform tags each customer by lifetime value. Predict the output, then run it.

✋ Predict first

The threshold is 10000. A customer with lifetime_value 80000 gets what tier? One with 4000?

Reveal the expected result

status,lifetime_value,is_active,tier
active,80000,true,gold
inactive,1200,false,standard
active,4000,true,standard

pipeline.yaml // editable

reset ⌖ explain ✓ dry-run ▶ run

pipeline: name: customer_etl nodes: - type: source name: customers config: name: customers type: csv path: ./data/customers.csv options: { has_header: true } schema: - { name: status, type: string } - { name: lifetime_value, type: string } - type: transform name: active_only input: customers config: cxl: | emit is_active = status == "active" - type: transform name: final_flag input: active_only config: cxl: | emit tier = if lifetime_value.to_int() > 10000 then "gold" else "standard" - type: output name: results input: final_flag config: name: results type: csv path: ./output/customers.csv

// quick check

Why is it `lifetime_value.to_int() > 10000` and not just `lifetime_value > 10000`?

• The value comes from a CSV as text; .to_int() makes it a number so the comparison is numeric
• to_int() makes the pipeline run faster
• Clinker requires every comparison to call a method first

Modify: add a field

Add a third column. After the final_flag transform computes tier, add one more emit line so the output also includes a greeting column equal to "Hello " + status.

✋ Predict first

You add emit greeting = 'Hello ' + status to the final_flag transform. For the active row, what is greeting?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

+ on text joins it, so "Hello " + "active" is "Hello active". A transform can emit as many columns as you like; each emit is one more column on the record.

Validate: catch a config mistake with --dry-run

Before you run a pipeline over real data, --dry-run reads the config and checks its structure and declared types, without processing a single record. This pipeline has a mistake of that kind: the schema declares lifetime_value as dollars, which is not a real type. Predict what --dry-run does, then run it.

✋ Predict first

The schema declares lifetime_value as type 'dollars'. Is that one of the types Clinker knows? What will --dry-run say, and will it exit cleanly?

Reveal the expected result

pipeline error in pipeline.yaml: config error: YAML parse error:
unknown variant `dollars`, expected one of null, bool, int, float,
string, date, date_time, array, map, numeric, any, nullable

(exit 1)

pipeline.yaml // editable

reset ⌖ explain ✓ dry-run ▶ run

pipeline: name: customer_etl nodes: - type: source name: customers config: name: customers type: csv path: ./data/customers.csv options: { has_header: true } schema: - { name: status, type: string } - { name: lifetime_value, type: dollars } - type: transform name: final_flag input: customers config: cxl: | emit tier = if lifetime_value.to_int() > 10000 then "gold" else "standard" - type: output name: results input: final_flag config: name: results type: csv path: ./output/customers.csv

--dry-run catches config and type mistakes before any data is read: a missing required field, or, as here, a schema type that doesn’t exist. The error names the valid types and the run stops with exit 1, so you fix it in a second instead of after a long job. (Fix it by declaring lifetime_value as string, the way the working pipeline above does.)

Not every mistake surfaces this early, though. --dry-run validates the config; it does not trace your data flow, so a logic error like pointing a node’s input: at a misspelled name isn’t caught here. That one only shows up when you actually run the pipeline.

Create: one pipeline, many deployments

Hard-coding 10000 means every team gets the same threshold. Instead, read it from a channel variable with $vars. The pipeline declares the variable; a channel file overrides it per deployment.

# customer_etl.yaml — declare the variable, then read it
pipeline:
  name: customer_etl
  vars:
    gold_threshold: { type: int, default: 10000 }
nodes:
  # ... source as before ...
  - type: transform
    name: final_flag
    input: customers
    config:
      cxl: |
        emit tier = if lifetime_value.to_int() > $vars.gold_threshold then "gold" else "standard"

# channels/acme-corp/customer_etl.channel.yaml — Acme wants a higher bar
channel:
  name: acme-corp
  target: ./customer_etl.yaml
vars:
  static:
    gold_threshold: { type: int, default: 50000 }

Run it with clinker run customer_etl.yaml --channel channels/acme-corp/customer_etl.channel.yaml, and Acme’s customers need 50000 to reach “gold”: same pipeline, no logic changed.

// quick check

What does a channel let you change about a pipeline?

• Its declared variables and settings, but not its logic (the nodes and CXL)
• Anything, including rewriting the transforms
• Only the output file name

Retrieval checkpoint

✓ Checkpoint

// quick check

A transform's CXL runs…

• once per record, top to bottom, with each emit adding a column
• once for the whole file
• only when an error occurs

// quick check

Back in lesson 01 you declared the schema, and here `lifetime_value` is declared as `string`. Why must the comparison call `.to_int()` before it checks `> 10000`?

• Because the type is declared, not inferred: a string isn't a number, so you convert it first
• Because .to_int() rounds the value before comparing
• Because Clinker re-reads the schema on every comparison

Common mistakes

• Comparing text to a number: "4000" > 10000 compares text. Convert first with .to_int().
• Using a type that doesn’t exist: a schema type: must be one Clinker knows; --dry-run rejects an unknown one with exit 1.
• Pointing input: at the wrong name: the input must be an exact node name:. --dry-run won’t notice this; it surfaces only when you run.

You should be able to:

• Write a transform that emits a new column with an if/then/else
• Explain why a CSV value needs .to_int() before a numeric comparison
• Read a --dry-run config error and fix an invalid schema type
• Move a hard-coded number into a $vars value overridden by a channel

Verify in the checkout:

clinker run customer_etl.yaml --dry-run
clinker run customer_etl.yaml --explain
clinker run customer_etl.yaml --channel channels/acme-corp/customer_etl.channel.yaml --dry-run

Go deeper (optional)

Curious how CXL is checked and run? Before your job processes records, Clinker parses your CXL and compiles it once. --dry-run validates the config’s structure and declared types; deeper logic, like a misspelled field or a value that won’t convert, is checked when the data actually flows. The Engine Track’s CXL: a staged language shows how. You don’t need it to write transforms; it’s there if you want it.