Transforms & CXL — compute new fields

The question for this lesson: your source gives you the columns that were in the file. But you almost always need new columns: a flag, a category, a cleaned-up value. How do you compute a new field for every record?

What you’ll be able to do

• Read and write a transform node that adds fields with CXL.
• Use emit, comparisons, if/then/else, and a method like .to_int().
• Pull a value out of a channel with $vars, so one pipeline serves many deployments.
• Validate a pipeline’s config and declared types with --dry-run before you run it.

This lesson assumes you’ve met the pipeline mental model and the source/transform/output nodes from What a pipeline is. If “record” or “node” feels fuzzy, start there.

Predict first

✋ Predict first

A transform contains: emit is_active = status == 'active'. The source row has status = 'active'. What new column appears, and what is its value? What about a row with status = 'churned'?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

emit adds a column named is_active. The expression status == "active" compares the status field to the text "active" and produces a true/false value. So the "active" row gets is_active = true; the "churned" row gets is_active = false.

A transform runs its CXL once per record, top to bottom, and each emit adds (or overwrites) one column on that record. That’s the whole idea. The rest is which expressions you write.

What CXL is

CXL is the little language inside a transform’s cxl: block. You write one statement per line. The ones you’ll use constantly:

Statement	What it does	Example
emit <name> = <expr>	add or overwrite a column	emit total = price * qty
let <name> = <expr>	a temporary value (not a column)	let net = price - discount
filter <condition>	drop rows where the condition is false	filter status == "active"
distinct	keep only unique rows	distinct by id

Expressions use the operators you’d expect (+ - * /, == != > <, and or not), plus if <cond> then <a> else <b>, and methods on values like .to_int(), .upper(), .trim().

🌱 New here? — why .to_int()?

This example deliberately declares lifetime_value as string, so the transform converts it before comparing it with a number. CSV does not force that choice: with type: int in the source schema, write lifetime_value > 10000 directly. Comparing a string with an integer without conversion is a type error, not a lexicographic numeric comparison. try_int() returns null on invalid text; it does not drop the row unless you explicitly filter that result.

Use: a real transform

This is the canonical customer_etl pipeline. The second transform tags each customer by lifetime value. Predict the output, then run it.

✋ Predict first

The threshold is 10000. A customer with lifetime_value 80000 gets what tier? One with 4000?

Reveal the expected result

status,lifetime_value,is_active,tier
active,80000,true,gold
inactive,1200,false,standard
active,4000,true,standard

pipeline.yaml // editable

reset ⌖ explain ✓ dry-run ▶ run

pipeline: name: customer_etl nodes: - type: source name: customers config: name: customers type: csv path: ./data/customers.csv options: { has_header: true } schema: - { name: status, type: string } - { name: lifetime_value, type: string } - type: transform name: active_only input: customers config: cxl: | emit is_active = status == "active" - type: transform name: final_flag input: active_only config: cxl: | emit tier = if lifetime_value.to_int() > 10000 then "gold" else "standard" - type: sink name: results input: final_flag config: name: results type: csv path: ./output/customers.csv

// quick check

Why is it `lifetime_value.to_int() > 10000` and not just `lifetime_value > 10000`?

• This schema deliberately declares the value as string; .to_int() converts it for numeric comparison
• to_int() makes the pipeline run faster
• Clinker requires every comparison to call a method first

Modify: add a field

Add a third column. After the final_flag transform computes tier, add one more emit line so the output also includes a greeting column equal to "Hello " + status.

✋ Predict first

You add emit greeting = 'Hello ' + status to the final_flag transform. For the active row, what is greeting?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

+ on text joins it, so "Hello " + "active" is "Hello active". A transform can emit as many columns as you like; each emit is one more column on the record.

Validate: catch a config mistake with --dry-run

Before you run a pipeline over real data, --dry-run reads the config and checks its structure and declared types, without processing a single record. This pipeline has a mistake of that kind: the schema declares lifetime_value as dollars, which is not a real type. Predict what --dry-run does, then run it.

✋ Predict first

The schema declares lifetime_value as type 'dollars'. Is that one of the types Clinker knows? What will --dry-run say, and will it exit cleanly?

Reveal the expected result

pipeline error in pipeline.yaml: config error: YAML parse error:
unknown variant `dollars`, expected one of null, bool, int, float, decimal,
string, date, date_time, array, map, numeric, any, nullable

(exit 1)

pipeline.yaml // editable

reset ⌖ explain ✓ dry-run ▶ run

pipeline: name: customer_etl nodes: - type: source name: customers config: name: customers type: csv path: ./data/customers.csv options: { has_header: true } schema: - { name: status, type: string } - { name: lifetime_value, type: dollars } - type: transform name: final_flag input: customers config: cxl: | emit tier = if lifetime_value.to_int() > 10000 then "gold" else "standard" - type: sink name: results input: final_flag config: name: results type: csv path: ./output/customers.csv

--dry-run catches config and type mistakes before any data is read: a missing required field, or, as here, a schema type that doesn’t exist. The error names the valid types and the run stops with exit 1, so you fix it in a second instead of after a long job. (Fix it by declaring lifetime_value as string, the way the working pipeline above does.)

Planning-only --dry-run also rejects a misspelled input:, unknown CXL field/method, or incompatible type. It cannot validate file contents without reading records; use a bounded preview or an isolated run for that.

Create: one pipeline, many deployments

Hard-coding 10000 means every team gets the same threshold. Instead, read it from a channel variable with $vars. The pipeline declares the variable; a channel file overrides it per deployment.

# customer_etl.yaml — declare the variable, then read it
pipeline:
  name: customer_etl
  vars:
    gold_threshold: { type: int, default: 10000 }
nodes:
  # ... source as before ...
  - type: transform
    name: final_flag
    input: customers
    config:
      cxl: |
        emit tier = if lifetime_value.to_int() > $vars.gold_threshold then "gold" else "standard"

# channel/acme-corp/customer_etl.channel.yaml — Acme wants a higher bar
channel:
  target: ../../customer_etl.yaml
vars:
  static:
    gold_threshold: { type: int, default: 50000 }

After creating the workspace catalog and channel manifest as shown in Channels, run it with clinker run customer_etl.yaml --channel acme-corp --base-dir ., and Acme’s customers need 50000 to reach “gold”: same pipeline, no logic changed.

// quick check

What does a channel let you change about a pipeline?

• Its declared variables and settings, but not its logic (the nodes and CXL)
• Anything, including rewriting the transforms
• Only the output file name

Retrieval checkpoint

✓ Checkpoint

// quick check

A transform's CXL runs…

• once per record, top to bottom, with each emit adding a column
• once for the whole file
• only when an error occurs

// quick check

Back in lesson 01 you declared the schema, and here `lifetime_value` is declared as `string`. Why must the comparison call `.to_int()` before it checks `> 10000`?

• Because the type is declared, not inferred: a string isn't a number, so you convert it first
• Because .to_int() rounds the value before comparing
• Because Clinker re-reads the schema on every comparison

Common mistakes

• Comparing text to a number: "4000" > 10000 is a type error. Declare a numeric source type, or explicitly convert deliberately retained text.
• Using a type that doesn’t exist: a schema type: must be one Clinker knows; --dry-run rejects an unknown one with exit 1.
• Pointing input: at the wrong name: the input must be an exact node name:. Planning-only --dry-run rejects a dangling input before execution.

You should be able to:

• Write a transform that emits a new column with an if/then/else
• Explain when a declared string needs .to_int(), and why a declared int does not
• Read a --dry-run config error and fix an invalid schema type
• Move a hard-coded number into a $vars value overridden by a channel

Verify in the checkout:

clinker run customer_etl.yaml --dry-run
clinker run customer_etl.yaml --explain
clinker run customer_etl.yaml --channel acme-corp --base-dir . --dry-run

Go deeper (optional)

Curious how CXL is checked and run? Before your job processes records, Clinker parses your CXL and compiles it once. --dry-run validates the config’s structure and declared types; deeper logic, like a misspelled field or a value that won’t convert, is checked when the data actually flows. The Engine Track’s CXL: a staged language shows how. You don’t need it to write transforms; it’s there if you want it.