Wiring & routing — connect nodes, split records

The question for this lesson: so far your pipelines have been a straight line: source, transform, output. But real work branches. Send big orders one way, everything else another. How do you build a pipeline that isn’t a straight line?

What you’ll be able to do

• Read how input: wires one node to another into a graph.
• Use a route node to split records down named branches.
• Read the shape of any pipeline by following the wiring.

Nodes are wired by name

Every node has a name:. Every non-source node has an input: naming the node it reads from. That’s the whole wiring system: Clinker builds the graph by matching input: values to node name:s.

- type: transform
  name: clean         # this node is called "clean"
  input: customers    # it reads from the node named "customers"

🌱 New here? — a DAG

A pipeline is a “DAG”, a directed acyclic graph. “Directed” = data flows one way (input → output). “Acyclic” = no loops; a node can’t eventually feed back into itself. You don’t have to draw it. Just follow each input: to see the shape.

Splitting records: the route node

A route node sends each record down one of several named branches, based on a condition. Downstream nodes read a branch with input: <route_name>.<branch>.

- type: route
  name: by_size
  input: orders
  config:
    mode: exclusive
    conditions:
      big: "amount.to_int() > 100"   # this branch gets the big orders
    default: small                    # everything else goes here

A record with amount over 100 goes down the big branch; everything else goes down small. Then two outputs read the two branches.

Predict first

✋ Predict first

A route has conditions { big: amount.to_int() > 100 } and default: small. Three orders arrive with amount 250, 80, and 100. Which branch does each take?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

• 250 → big (250 > 100 is true).
• 80 → small (the default; 80 > 100 is false).
• 100 → small. Watch the boundary: the condition is greater than 100, and 100 is not greater than 100, so it falls through to the default.

Boundary conditions (> vs >=) are the most common routing bug, so always test the exact edge.

Use: a fan-out pipeline

Predict which file each order lands in, then run it.

✋ Predict first

orders have amount 250, 80, and 100. With branch big = amount.to_int() > 100, which rows go to high.csv and which to low.csv?

Reveal the expected result

high.csv:
id,amount
A,250

low.csv:
id,amount
B,80
C,100

pipeline.yaml // editable

reset ⌖ explain ✓ dry-run ▶ run

pipeline: name: route_fanout nodes: - type: source name: orders config: name: orders type: csv path: ./data/orders.csv options: { has_header: true } schema: - { name: id, type: string } - { name: amount, type: string } - type: route name: by_size input: orders config: mode: exclusive conditions: high: "amount.to_int() > 100" default: low - type: output name: high input: by_size.high # reads the 'high' branch config: name: high type: csv path: ./output/high.csv - type: output name: low input: by_size.low # reads the 'low' branch config: name: low type: csv path: ./output/low.csv

// quick check

How does the `high` output node get only the big orders?

• It reads input: by_size.high, the named branch of the route
• It re-checks the amount itself
• The route writes the file directly

Modify: add a third branch

Change the route so orders over 1000 go to a new vip branch, over 100 stay high, and the rest are low. You’ll add a vip condition and a third output reading by_size.vip.

Create: route by category, not just amount

The Modify split on a number. Now author a route from scratch that splits on a category, a string field. Send each order to a per-region output:

- type: route
  name: by_region
  input: orders
  config:
    mode: exclusive
    conditions:
      us: "region == 'US'"      # CXL string equality, not a threshold
      eu: "region == 'EU'"
    default: other              # everything else

Three outputs then read the three branches: by_region.us, by_region.eu, and by_region.other. Over rows A(US), B(EU), C(US), D(APAC):

us.csv:    A, C
eu.csv:    B
other.csv: D        # APAC matched no condition, so it took the default

A route condition is any CXL predicate (> on a number or == on a string), and mode: exclusive sends each record to the first branch it matches, or to default if none. Confirm the branch wiring with clinker run your_pipeline.yaml --explain.

Retrieval checkpoint

✓ Checkpoint

// quick check

The route's branch condition is `amount.to_int() > 100`. From lesson 02: which little language is that condition written in, and why the `.to_int()`?

• CXL, and `.to_int()` is there because `amount` is a declared string, which you convert before comparing as a number
• YAML, and `.to_int()` just speeds up the route
• SQL, and `.to_int()` is required on every route condition

Common mistakes

• Off-by-one at the boundary: decide whether the edge value belongs in the branch (> vs >=).
• Reading the route, not its branch: downstream you read route.branch, not just route.

You should be able to:

• Trace a pipeline's shape by following each input:
• Write a route with a condition branch and a default
• Read a branch downstream with input: <route>.<branch>
• Explain why an order of exactly 100 took the default branch

Verify in the checkout:

clinker run route_fanout.yaml --dry-run
clinker run route_fanout.yaml --explain

Go deeper (optional)

Curious how Clinker turns your nodes: list into an execution plan and runs the branches? That’s the Engine Track’s Through the DAG. Not needed to build pipelines, but there if you want it.