Operating a pipeline — run it for real

The question for this lesson: your pipeline is built and validated. Now you have to operate it: run it on real volumes without exhausting memory, know whether it actually succeeded, and run it on a schedule. How?

What you’ll be able to do

• Bound a run’s memory with --memory-limit (and let Clinker spill to disk under pressure).
• Read the exit code to tell a clean run from a partial one from a failure.
• Find a run’s metrics and logs.
• Schedule a pipeline with external tools, because Clinker has no built-in scheduler.

Clinker is a finite batch job

This is the mental model the whole lesson rests on: Clinker reads its inputs, processes them, writes its outputs, and exits. It is not a server and not a daemon; there is no “running Clinker” sitting in the background waiting. Every run is one finite job with a beginning and an end, exactly like grep or a backup script. That single fact decides how you bound it, watch it, and schedule it.

Bound the memory; spill when it’s tight

Clinker accounts retained state and responds to pressure with admission refusal, backpressure, spilling, or a controlled failure. Spilling helps when an operator can release enough memory; it does not guarantee that every workload fits every budget. Source decoding, writer preparation, irreducible state, and the process baseline still need space.

clinker run customer_etl.yaml --memory-limit 256M --threads 4

The default memory limit is 512M. You can also set it in the pipeline:

pipeline:
  name: customer_etl
  memory: { limit: "256M", backpressure: spill }

🌱 New here? — backpressure: spill

The spill policy prefers freeing retained records to disk. A producer-pausing policy also needs a satisfiable budget above measured baseline RSS; otherwise it could wait forever for memory the process cannot release. Spill-only mode can attempt progress below that baseline, but can still fail to admit required state. Neither mode turns the configured limit into an OS-enforced RSS ceiling.

✋ Predict first

If you shrink a pipeline's memory limit to 16K and select spill, must it finish?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

No. Some retained state can spill, but decoding, bookkeeping, and an individual operation can require more than the remaining budget. The engine may refuse or abort the run. A tiny fixture demonstrates a pressure path; it is not evidence that every input shape fits. Inspect the actual outcome and diagnostics.

Read the exit code

When the run ends, its exit code is how a script (or you) know what happened. The ones to recognize:

• 0 is clean completion. Filters and routing may intentionally omit rows.
• 2 means completed, but some rows were dead-lettered (the DLQ from lesson 07). The job did its work; you just have rows to review. Whether that is acceptable depends on your data-quality policy.
• 1 is a failure: a config/schema error, a fail-fast error, or the memory budget was blown.
• 3 is too many bad rows (a DLQ-rate limit) or a CXL evaluation error; 4 is an I/O, format, or spill error; 130 is interrupted (Ctrl-C / SIGTERM).

✋ Predict first

You run a pipeline with error_handling strategy: continue and a dlq: configured (from lesson 07). 3 of 10,000 rows fail to parse and land in the DLQ; the rest are written. What exit code do you get, and should your nightly job treat it as a failure?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

Exit code 2, completed with DLQ. The run succeeded: 9,997 rows were written and 3 were set aside for review. A wrapper should distinguish completed-with-DLQ from an aborted run. Your data-quality policy can still reject a completed run that lost records; do not treat exit 2 as interchangeable with clean success. Configure the declared source-type rejection ratio with error_handling.type_error_threshold in YAML. This ratio excludes unrelated DLQ reasons; --error-threshold is retired.

Find the metrics and logs

To see how a run behaved (rows in/out per node, timings), spool metrics to a directory and collect them:

clinker run customer_etl.yaml --metrics-spool-dir ./metrics
clinker metrics collect ./metrics

For more detail while it runs, raise the log level:

clinker run customer_etl.yaml --log-level debug

Schedule it: with your own tools

Because Clinker is a finite batch job with no built-in scheduler or daemon, you don’t ask Clinker to “run nightly.” You run it once, and let your operating system or orchestrator do the scheduling: cron, a systemd timer, or Airflow.

# crontab — run the customer ETL every night at 02:00
0 2 * * *  clinker run /etc/clinker/customer_etl.yaml >> /var/log/customer_etl.log 2>&1

That’s the whole pattern: a normal command, on a normal schedule, with its exit code and logs captured like any other batch job.

Retrieval checkpoint

✓ Checkpoint

// quick check

Which exit codes indicate completion rather than an aborted run?

• 0 (clean) and 2 (completed, some rows dead-lettered)
• Only 0; any non-zero code is a failure
• 0, 1, and 2; only 3 and 4 are real failures

// quick check

From lesson 07: exit code 2 specifically means what about the run?

• It completed, but some rows were routed to the dead-letter queue under a continue strategy
• The configuration was invalid and nothing ran
• The process was interrupted partway

Common mistakes

• Looking for a daemon/service mode. There isn’t one. Clinker runs once and exits; schedule it externally (cron/systemd/Airflow).
• Treating exit code 2 as clean success. It means completed with dead-lettered rows; your data-quality policy decides whether downstream work should continue.
• Assuming spilling guarantees completion. Set a realistic memory budget and inspect admission, spill, and failure diagnostics.

You should be able to:

• Set a memory budget with --memory-limit, and say what backpressure: spill does
• Map exit codes 0 / 2 / 1 to clean / completed-with-DLQ / failed
• Spool and collect a run's metrics
• Schedule a pipeline with cron (and say why Clinker itself has no scheduler)

Verify in the checkout:

clinker run customer_etl.yaml --memory-limit 256M --dry-run
clinker run customer_etl.yaml --metrics-spool-dir ./metrics

Go deeper (optional)

Bounded memory + spill is why Clinker can process a file far larger than RAM on a small box. The Engine Track’s Bounded memory in action shows the machinery. Next: the file formats Clinker reads and writes, from CSV to EDI.