Operating a pipeline — run it for real

The question for this lesson: your pipeline is built and validated. Now you have to operate it: run it on real volumes without exhausting memory, know whether it actually succeeded, and run it on a schedule. How?

What you’ll be able to do

• Bound a run’s memory with --memory-limit (and let Clinker spill to disk under pressure).
• Read the exit code to tell a clean run from a partial one from a failure.
• Find a run’s metrics and logs.
• Schedule a pipeline with external tools, because Clinker has no built-in scheduler.

Clinker is a finite batch job

This is the mental model the whole lesson rests on: Clinker reads its inputs, processes them, writes its outputs, and exits. It is not a server and not a daemon; there is no “running Clinker” sitting in the background waiting. Every run is one finite job with a beginning and an end, exactly like grep or a backup script. That single fact decides how you bound it, watch it, and schedule it.

Bound the memory; spill when it’s tight

A batch job over a big file could try to hold too much in memory. Clinker won’t: you give it a memory budget, and when a buffering step (an aggregate, a reshape) would exceed it, Clinker spills the overflow to disk and keeps going instead of crashing.

Set the budget on the command line:

clinker run customer_etl.yaml --memory-limit 256M --threads 4

--memory-limit defaults to 512M; --threads sets how much runs in parallel. Or pin the budget in the pipeline itself. This is Clinker’s real scd_type2 example, which sets a tiny budget on purpose so even a small run exercises the spill path:

pipeline:
  name: scd_type2_backfill
  memory: { limit: "16K", backpressure: spill }   # tight budget → spill to disk

🌱 New here? — backpressure: spill

backpressure says what to do when a step fills its memory budget. spill writes the overflow to a temp file and reads it back: slower, but the run finishes in bounded memory. (The other modes pause or do both.) The point: a job too big for RAM still completes.

Now change it. Take that comfortable --memory-limit 256M and crank the budget down to a deliberately tiny --memory-limit 16K, or set it in the pipeline the way the scd_type2 example does:

pipeline:
  memory: { limit: "16K", backpressure: spill }   # far below what the data needs

✋ Predict first

A buffering step (an aggregate) needs more than 16K of working memory to hold its data, but you've capped the budget at 16K with backpressure: spill. Does the run still complete, and why?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

Yes, it still completes. The aggregate hits the 16K ceiling, and backpressure: spill sends the overflow to a temp file on disk and reads it back as needed. The run is slower (disk is slower than RAM), but it stays inside its memory budget and finishes. That’s the whole reason Clinker’s real scd_type2 example pins such a tiny budget: 16K is below what even a small dataset needs, so the spill path is guaranteed to engage. It’s a way to exercise spill on purpose. The run would only break (exit 4) if the spill itself failed: no temp space, a disk I/O error. Shrinking the budget doesn’t fail the job; it just forces it onto disk.

Read the exit code

When the run ends, its exit code is how a script (or you) know what happened. The ones to recognize:

• 0 is clean success. Every row made it through.
• 2 means completed, but some rows were dead-lettered (the DLQ from lesson 07). The job did its work; you just have rows to review. This is not a failure.
• 1 is a failure: a config/schema error, a fail-fast error, or the memory budget was blown.
• 3 is too many bad rows (a DLQ-rate limit) or a CXL evaluation error; 4 is an I/O, format, or spill error; 130 is interrupted (Ctrl-C / SIGTERM).

✋ Predict first

You run a pipeline with error_handling strategy: continue and a dlq: configured (from lesson 07). 3 of 10,000 rows fail to parse and land in the DLQ; the rest are written. What exit code do you get, and should your nightly job treat it as a failure?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

Exit code 2, completed with DLQ. The run succeeded: 9,997 rows were written and 3 were set aside for review. A wrapper script that treats any non-zero code as failure would wrongly alarm on this. The right rule is usually: 0 and 2 are “ran”; 1, 3, 4 are “broke.” (How many DLQ’d rows is too many is the --error-threshold knob.)

Find the metrics and logs

To see how a run behaved (rows in/out per node, timings), spool metrics to a directory and collect them:

clinker run customer_etl.yaml --metrics-spool-dir ./metrics
clinker metrics collect ./metrics

For more detail while it runs, raise the log level:

clinker run customer_etl.yaml --log-level debug

Schedule it: with your own tools

Because Clinker is a finite batch job with no built-in scheduler or daemon, you don’t ask Clinker to “run nightly.” You run it once, and let your operating system or orchestrator do the scheduling: cron, a systemd timer, or Airflow.

# crontab — run the customer ETL every night at 02:00
0 2 * * *  clinker run /etc/clinker/customer_etl.yaml >> /var/log/customer_etl.log 2>&1

That’s the whole pattern: a normal command, on a normal schedule, with its exit code and logs captured like any other batch job.

Retrieval checkpoint

✓ Checkpoint

// quick check

A nightly wrapper script should treat which exit codes as 'the job ran successfully enough to continue'?

• 0 (clean) and 2 (completed, some rows dead-lettered)
• Only 0; any non-zero code is a failure
• 0, 1, and 2; only 3 and 4 are real failures

// quick check

From lesson 07: exit code 2 specifically means what about the run?

• It completed, but some rows were routed to the dead-letter queue under a continue strategy
• The configuration was invalid and nothing ran
• The process was interrupted partway

Common mistakes

• Looking for a daemon/service mode. There isn’t one. Clinker runs once and exits; schedule it externally (cron/systemd/Airflow).
• Treating exit code 2 as a failure. It means “completed, with dead-lettered rows,” not “broke.”
• Assuming a big job will OOM. Set --memory-limit (or pipeline.memory) and let it spill; the run stays bounded.

You should be able to:

• Set a memory budget with --memory-limit, and say what backpressure: spill does
• Map exit codes 0 / 2 / 1 to clean / completed-with-DLQ / failed
• Spool and collect a run's metrics
• Schedule a pipeline with cron (and say why Clinker itself has no scheduler)

Verify in the checkout:

clinker run customer_etl.yaml --memory-limit 256M --dry-run
clinker run customer_etl.yaml --metrics-spool-dir ./metrics

Go deeper (optional)

Bounded memory + spill is why Clinker can process a file far larger than RAM on a small box. The Engine Track’s Bounded memory in action shows the machinery. Next: the file formats Clinker reads and writes, from CSV to EDI.