Joining streams — combine & merge

The question for this lesson: so far each pipeline has read from one source. But real data is scattered. Orders live in one file, the product catalog in another; web logins in one stream, mobile logins in a second. How do you bring two streams together?

There are two different ways, and choosing the right one is the whole lesson:

• combine (join two streams on a key) makes each row wider (orders + their product details).
• merge (stack same-shaped streams) makes more rows (web logins + mobile logins, one list).

Run these in a clinker checkout

A join reads two data files at once, which the in-page sandbox doesn’t carry, so this lesson teaches from Clinker’s real example pipelines and asks you to predict their output. Run them for real in a clinker checkout with clinker run … --dry-run / --explain.

What you’ll be able to do

• Use a combine node to join two streams on a key, and say what match and on_miss do.
• Use a merge node to stack same-shaped streams into one.
• Say when you need combine vs merge.

Combine: join two streams on a key

A combine node takes two named inputs and joins them where a key matches. This is Clinker’s real order_fulfillment join: orders enriched with details from a product catalog.

- type: combine
  name: product_lookup
  input:
    orders: normalize_fields        # two NAMED inputs (not a list)
    products: products
  config:
    where: "orders.product_code == products.product_code"   # the join key
    match: first                    # if several products match, take the first
    on_miss: null_fields            # no match? keep the order, product fields null
    cxl: |
      emit order_id     = orders.order_id
      emit product_name = products.product_name
      emit category     = products.category

🌱 New here? — input: as a named map

A normal node has a single input: naming one upstream node (lesson 03). A combine instead takes a named map of inputs (orders: and products:), so its where and cxl can say which side a field comes from (orders.order_id vs products.product_name). The names are yours to choose; they’re just labels for the two sides.

The two knobs that decide the join’s behavior:

• match: when more than one row on the other side matches the key, first takes one; other modes keep all matches.
• on_miss: when nothing matches, null_fields keeps the order and leaves the product columns null (a left join). The alternative is to drop the unmatched row.

Predict first

✋ Predict first

An order has product_code 'A1'. The products stream has A1 → {name: Widget, category: Hardware} and B2 → {name: Gadget, category: Toys}. With where: orders.product_code == products.product_code and on_miss: null_fields, what does the combined row for that order contain? And what if the order's product_code were 'Z9' (not in products)?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

For A1: the order keys to the A1 product, so the combined row carries the order’s fields plus product_name = "Widget" and category = "Hardware".

For Z9: nothing matches. Because on_miss: null_fields, the order is kept but product_name and category come out null. You don’t lose the order, you just have no catalog detail for it. (Had on_miss dropped misses instead, the Z9 order would vanish, a common way to silently lose rows.)

Merge: stack same-shaped streams

When two streams are the same shape and you just want them in one list, use merge. This is Clinker’s multi_source_session example: web and mobile logins, combined into one stream.

- type: merge
  name: all_logins
  inputs: [src_web, src_mobile]     # a LIST — order-agnostic union

No where, no key. merge simply interleaves the rows of every input into a single stream for the next node (often an aggregate). Note the shape of the wiring is different from combine: merge takes inputs: as a list, because the streams are peers; combine takes input: as a named map, because it must tell the two sides apart.

// quick check

You have last year's sales and this year's sales in two files, same columns, and want one combined dataset to total. combine or merge?

• merge, because the streams are the same shape; you want more rows, not wider rows
• combine, because you always combine two streams with combine
• Neither; you must concatenate the files by hand first

Combining streams isn’t the only thing you can do across rows. Restructuring rows (rewriting some of them, or dropping/routing whole groups) gets its own lesson next, with the reshape and cull nodes.

Retrieval checkpoint

✓ Checkpoint

// quick check

From lesson 03: a normal node wires its single upstream with `input: <name>`. How is a combine node's wiring different, and why?

• combine uses a named map (input: { orders: …, products: … }) so its where/cxl can tell the two sides apart
• combine uses input: <name> exactly like every other node
• combine needs no input because it reads files directly

Common mistakes

• Using merge to join on a key. merge only stacks rows; it has no where. To match rows across streams by a key, you need combine.
• Forgetting on_miss. Decide what an unmatched row should do. null_fields keeps it (left join); dropping it can silently lose data.
• Mismatched shapes in a merge. merge expects its inputs to share a shape; stacking unlike rows leads to surprises downstream.

You should be able to:

• Use a combine node with two named inputs and a where key
• Say what match: first and on_miss: null_fields do
• Use a merge node to stack same-shaped streams
• Choose combine vs merge for a given task

Verify in the checkout:

clinker run order_fulfillment.yaml --dry-run
clinker run order_fulfillment.yaml --explain

Go deeper (optional)

combine makes rows wider (join), merge makes more rows (union). Keep those two verbs straight and most “bring it together” tasks fall into place. Next: restructuring rows in place. reshape rewrites rows within a group, and cull drops or side-routes whole groups.