Channels — one pipeline, many tenants

The question for this lesson: you have one customer_etl pipeline, and several customers (tenants) want the same logic with one number changed. Acme Corp only calls someone “gold” at a much higher lifetime value than everyone else. Do you copy the whole pipeline and edit a line for each tenant? No. You keep one pipeline and add a small channel overlay per tenant.

What you’ll be able to do

Read a channel overlay and say what it overrides, and what it can’t.
Override a pipeline’s declared variable for one deployment, without touching its logic.
Apply a channel at run time with --channel.

A channel overrides values, not logic

A channel is a small overlay file that sits on top of a base pipeline and changes its declared variables and config knobs for one deployment, never its nodes, never its CXL. One customer_etl.yaml, many tenants.

It works in two halves. First, the base pipeline declares a variable with a default:

pipeline:
  name: customer_etl
  vars:
    gold_threshold: { type: int, default: 10000 }   # the lifetime-value cut for "gold"

and reads it in a CXL expression with $vars:

emit tier = if lifetime_value.to_int() > $vars.gold_threshold then "gold" else "standard"

🌱 New here? — $vars

$vars.gold_threshold reads the value of the gold_threshold variable declared at the top of the pipeline. Declaring a number once, by name, instead of hardcoding 10000 inside the expression, is exactly what lets a channel change it later without editing the logic.

Then a channel overlay supplies a different value for one tenant:

channel:
  name: acme-corp
  target: ./customer_etl.yaml      # the base pipeline this overlays
vars:
  static:
    gold_threshold:
      type: int
      default: 50000               # Acme's higher bar for "gold"

and you apply it at run time with --channel:

clinker run customer_etl.yaml --channel channels/acme-corp/customer_etl.channel.yaml

Same nodes, same CXL, same everything: only gold_threshold changes, and only for Acme. A channel can override declared vars: and config: knobs; it can not add or rewire nodes, or change an expression. If two tenants need different logic, that’s a different pipeline, not a channel.

Use: run the base pipeline

First, the base on its own, with no channel, so gold_threshold is its declared default, 10000. Predict each customer’s tier, then run.

pipeline.yaml // editable

pipeline:
  name: customer_etl
  vars:
    # The lifetime-value cut for the "gold" tier. Declared here with a default;
    # a channel overlay can override it per tenant without touching the logic.
    gold_threshold: { type: int, default: 10000 }
nodes:
  - type: source
    name: customers
    config:
      name: customers
      type: csv
      path: ./data/customers.csv
      options: { has_header: true }
      schema:
        - { name: status, type: string }
        - { name: lifetime_value, type: string }

  - type: transform
    name: active_only
    input: customers
    config:
      cxl: |
        emit is_active = status == "active"

  - type: transform
    name: final_flag
    input: active_only
    config:
      cxl: |
        emit tier = if lifetime_value.to_int() > $vars.gold_threshold then "gold" else "standard"

  - type: output
    name: results
    input: final_flag
    config:
      name: results
      type: csv
      path: ./output/customers.csv

> output appears here — predict, then run

Only the 80000 customer clears 10000, so only they are gold.

Predict first: now apply Acme’s channel

Acme’s channel sets gold_threshold to 50000. The pipeline is identical; only that one number changes.

Create: a channel for a generous tenant

Now write your own. Bright Start is a tenant that rewards smaller customers: anyone over 3000 lifetime value should count as gold. Write its channel file.

✋ Predict first

Write channels/bright-start/customer_etl.channel.yaml so gold_threshold becomes 3000. Which of the three customers above (80000, 1200, 4000) is gold under Bright Start?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

channel:
  name: bright-start
  target: ./customer_etl.yaml
vars:
  static:
    gold_threshold:
      type: int
      default: 3000

Run it with clinker run customer_etl.yaml --channel channels/bright-start/customer_etl.channel.yaml. Now both the 80000 and the 4000 customer clear 3000 and come out gold (only the 1200 customer stays standard), and you changed exactly one number, in a file that never touches the pipeline’s logic.

// quick check

What does Acme's channel actually change in the pipeline?

Retrieval checkpoint

✓ Checkpoint

// quick check

From lesson 02: the tier is computed with `lifetime_value.to_int() > $vars.gold_threshold`. Why the `.to_int()`?

Common mistakes

Trying to change logic in a channel: a channel only overrides declared vars:/config:. Rewiring nodes or editing CXL belongs in the base pipeline (or a new one).
Overriding a variable the base never declared: a channel can only override a value that already exists in the base pipeline.vars. Declare it first.
Off-by-one at the boundary: a value exactly equal to the threshold is not greater than it.

You should be able to:

Say what a channel can override (declared vars/config) and what it can't (logic/nodes)
Read $vars.gold_threshold and trace where its value comes from
Write a channel that overrides a declared variable for one tenant
Apply a channel at run time with --channel

Verify in the checkout:

clinker run customer_etl.yaml --channel channels/acme-corp/customer_etl.channel.yaml --dry-run
clinker run customer_etl.yaml --channel channels/acme-corp/customer_etl.channel.yaml --explain