Error handling

A pipeline chewing through ten million records will hit failures of very different weight. One malformed row in a CSV is data: annoying, expected, and no reason to throw away the other 9,999,999 records. A disk filling mid-write is infrastructure. And an invariant the compiler proved at plan time being violated at run time is a bug in clinker itself, which must stop everything, loudly. A good error type doesn’t just say that something failed; it encodes which kind, because the kind decides the fate of the run.

Prerequisites

• The plan/runtime boundary

→

Leads to

• Building an error type by hand

What you’ll be able to do

• Read the PipelineError enum and name the subsystem each variant aggregates.
• Explain how a From<E> impl lets the ? operator lift a foreign subsystem error into PipelineError.
• Distinguish a recoverable per-record error from a fatal one, and say what each does under ErrorStrategy::Continue.
• Trace a per-record eval error through the dispatch router to the dead-letter queue, and say why Internal never reaches that router.

New terms in this lesson (each is also expanded inline at the point you first need it):

• error strategy: the policy (FailFast vs Continue) that decides a recoverable error’s fate.
• recoverable: a per-record data error (a cell that won’t cast, one bad row) that a non-fail-fast policy may quarantine and keep going.
• fatal: an error that aborts the run regardless of policy. Internal, MemoryBudgetExceeded, schema/sort-order violations.
• DLQ (dead-letter queue): where quarantined records go so one bad row doesn’t sink the job.
• the ? operator: one-character early-return-on-error that converts the error via From; you met it in Missing cells & fallible parses, re-used here.

Predict first

✋ Predict first

A job runs under ErrorStrategy::Continue. Row 5 fails to coerce a text cell to an integer (a per-record Eval error). Row 6 trips an Internal error, a plan-time invariant violated at run time. Which of these two does the run survive, and which kills it? Write down your call for each before revealing.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

The job survives row 5 and dies on row 6, and that asymmetry is the whole point of this lesson.

• Row 5’s Eval error is recoverable. Continue means “quarantine bad rows to the DLQ and keep going,” so row 5’s record is set aside and processing carries on.
• Row 6’s Internal error is fatal. Its doc says “ALWAYS aborts the run regardless of ErrorStrategy::Continue.” The run stops, loudly, no matter the policy.

The wrong mental model is “Continue means continue through everything.” It does not. Continue is a statement about your data (messy rows are tolerable), never a license to limp on through a broken engine. A clinker bug is not a bad record, and routing it to the DLQ would hide the bug behind a “bad row” label. Hold that split: recoverable continues, fatal always aborts; everything below earns it.

One enum that aggregates every failure

PipelineError is the engine’s top-level runtime error. It’s a sum type with one variant per subsystem failure, plus a set of specific “this went wrong” variants:

clinker-plan ·error.rs ·PipelineError type @19acdcb4

#[derive(Debug)]
pub enum PipelineError {
    Config(crate::config::ConfigError),
    Format(clinker_format::FormatError),
    Eval(cxl::eval::EvalError),
    Io(std::io::Error),
    /// Plan-time invariant violated at runtime — Clinker bug, not a data
    /// error. ALWAYS aborts the run regardless of ErrorStrategy::Continue.
    Internal { op: &'static str, node: String, detail: String },
    MemoryBudgetExceeded { node: String, used: u64, limit: u64, /* … */ },
    // ... ~25 variants; many documented "Always aborts the run"
}

Notice this enum is hand-written, with no thiserror derive. error.rs writes its own Display, its own impl std::error::Error, and its own From conversions. That’s a deliberate choice for a type this central: the Display strings are diagnostics shown to users, worth hand-tuning. (You’ll meet thiserror elsewhere in the codebase; it’s a fine tool, merely not used for this type.)

From impls are what make ? work

The central piece of idiomatic Rust error handling is ?. Writing let r = next_record()?; means “if this is Err, return it from the enclosing function, converting it to my error type on the way out.” That conversion is powered by the From trait. Each subsystem error gets a hand-written From into PipelineError:

impl From<std::io::Error> for PipelineError {
    fn from(e: std::io::Error) -> Self { Self::Io(e) }
}
impl From<cxl::eval::EvalError> for PipelineError {
    fn from(e: cxl::eval::EvalError) -> Self { Self::Eval(e) }
}
// ... ConfigError, FormatError, SchemaError, SpillError

With those in place, a function returning Result<_, PipelineError> can call into the format layer, the IO layer, and the CXL evaluator and write ? after each. Every foreign error is auto-lifted into the right PipelineError variant. No match, no manual map_err. The six From impls are the seams that let one error type absorb six others.

The taxonomy: abort versus quarantine

Here’s where the type earns its keep. The variants split into two fates:

• Recoverable (per-record data errors). A Value that won’t cast, a cxl::eval::EvalError on one row. Under the right policy these are sent to a dead-letter queue (DLQ): the bad record is set aside and the run continues.
• Fatal (always abort). Internal, MemoryBudgetExceeded, schema mismatches, sort-order violations. These stop the run regardless of policy. The variant docs say so in capital letters: “ALWAYS aborts the run regardless of ErrorStrategy::Continue.”

Which policy governs the recoverable ones is a config setting, the error strategy:

clinker-plan ·pipeline.rs ·ErrorStrategy type @19acdcb4

pub enum ErrorStrategy {
    FailFast,    // any error stops the run (the default)
    Continue,    // recoverable errors go to the DLQ; keep going
    BestEffort,
}

And the actual routing, the place a per-record eval error meets the policy, is one function in the executor:

clinker-exec ·dispatch.rs ·dispatch_transform_eval_error fn @19acdcb4

crates/clinker-exec/src/executor/dispatch.rs

fn dispatch_transform_eval_error(/* … */) -> Result<…, PipelineError> {
    if ctx.strategy == ErrorStrategy::FailFast {
        return Err(eval_err.into());     // propagate — the ? at the call site aborts
    }
    // otherwise: classify and route the bad record to the DLQ, run continues
}

The crucial asymmetry: only recoverable errors flow through this router. The fatal variants are never offered to the DLQ at all. Internal and its kin are constructed and ?-propagated directly, bypassing this function entirely. So ErrorStrategy::Continue can keep a job alive through a million bad rows, but it cannot suppress a clinker bug. That’s by design: a Continue policy is a statement about your data, never a license to limp on through a broken engine.

Why Internal is its own fate

Pause on Internal { op, node, detail }. Its doc calls it “a Clinker bug, not a data error.” It exists for the cases that the plan/runtime boundary (last lesson) was supposed to make impossible: an invariant that compilation proved, tripped anyway at run time. Routing such a thing to the DLQ would be exactly wrong: it would hide a bug behind a “bad record” label and let a broken run produce plausible-looking output. Making Internal always-fatal means the engine fails loudly at the first sign it has violated its own contract. The run dies, the operator sees it, nobody trusts corrupt output. The error taxonomy is how the engine refuses to paper over its own bugs.

Worked → completion → faded

Now you build the taxonomy in miniature (?, From, and the abort/quarantine split), scaffolded down to your own code. Each rung adds exactly one new idea: first read the worked machine, then complete the routing decision, then write the From-powered propagation from scratch.

Worked: the whole taxonomy, annotated

Here is a self-contained model of the engine’s split, all in std. Run it and watch the two classes diverge: Internal aborts under every policy, Data only aborts under fail-fast.

rust // editable

reset ▶ run

#[derive(Debug)]
enum DataError { BadField(String) }        // recoverable: one bad record

#[derive(Debug)]
enum PipelineError {
  Data(DataError),                       // recoverable — may go to the DLQ
  Internal(&'static str),                // a bug — always fatal
}

// From lets `?` convert DataError into PipelineError automatically.
impl From<DataError> for PipelineError {
  fn from(e: DataError) -> Self { PipelineError::Data(e) }
}

fn parse_age(s: &str) -> Result<u32, DataError> {
  s.parse().map_err(|_| DataError::BadField(format!("not a number: {s}")))
}

fn process(s: &str) -> Result<u32, PipelineError> {
  let age = parse_age(s)?;        // From<DataError> fires here — no manual map_err
  if age > 200 {
      // 'impossible' — treat as a bug, not as bad data
      return Err(PipelineError::Internal("age over 200 slipped past validation"));
  }
  Ok(age)
}

// The taxonomy decides each error's fate.
fn run(records: &[&str], fail_fast: bool) {
  for r in records {
      match process(r) {
          Ok(age) => println!("ok: {age}"),
          Err(PipelineError::Internal(msg)) => {
              println!("ABORT (bug): {msg}");
              return;             // internal errors always stop the run
          }
          Err(PipelineError::Data(e)) => {
              if fail_fast {
                  println!("ABORT (fail-fast): {e:?}");
                  return;
              }
              println!("quarantine to DLQ, keep going: {e:?}");
          }
      }
  }
}

fn main() {
  let records = ["42", "oops", "37"];
  println!("-- Continue: recoverable errors quarantined --");
  run(&records, false);
  println!("-- FailFast: first error aborts --");
  run(&records, true);
}

#[derive(Debug)] enum DataError { BadField(String) } // recoverable: one bad record #[derive(Debug)] enum PipelineError { Data(DataError), // recoverable — may go to the DLQ Internal(&'static str), // a bug — always fatal } // From lets `?` convert DataError into PipelineError automatically. impl From<DataError> for PipelineError { fn from(e: DataError) -> Self { PipelineError::Data(e) } } fn parse_age(s: &str) -> Result<u32, DataError> { s.parse().map_err(|_| DataError::BadField(format!("not a number: {s}"))) } fn process(s: &str) -> Result<u32, PipelineError> { let age = parse_age(s)?; // From<DataError> fires here — no manual map_err if age > 200 { // 'impossible' — treat as a bug, not as bad data return Err(PipelineError::Internal("age over 200 slipped past validation")); } Ok(age) } // The taxonomy decides each error's fate. fn run(records: &[&str], fail_fast: bool) { for r in records { match process(r) { Ok(age) => println!("ok: {age}"), Err(PipelineError::Internal(msg)) => { println!("ABORT (bug): {msg}"); return; // internal errors always stop the run } Err(PipelineError::Data(e)) => { if fail_fast { println!("ABORT (fail-fast): {e:?}"); return; } println!("quarantine to DLQ, keep going: {e:?}"); } } } } fn main() { let records = ["42", "oops", "37"]; println!("-- Continue: recoverable errors quarantined --"); run(&records, false); println!("-- FailFast: first error aborts --"); run(&records, true); }

Under Continue, "oops" is quarantined and "37" still processes; under FailFast, the first bad record stops everything. Either way, an Internal would abort, but in this version nothing triggers one yet. The next rung makes you supply that decision.

Completion: route the error to its fate

Below, the run loop is written except for the recoverable arm. The Internal arm is done for you; it always aborts. Complete the Data arm: under Continue (not fail-fast) a per-record data error is quarantined to the DLQ and the loop keeps going; under FailFast it aborts. One if decides which.

rust // editable

reset ▶ run

#[derive(Debug)]
enum DataError { BadField(String) }

#[derive(Debug)]
enum PipelineError {
  Data(DataError),
  Internal(&'static str),
}

impl From<DataError> for PipelineError {
  fn from(e: DataError) -> Self { PipelineError::Data(e) }
}

fn process(s: &str) -> Result<u32, PipelineError> {
  let age: u32 = s.parse().map_err(|_| DataError::BadField(format!("not a number: {s}")))?;
  if age > 200 {
      return Err(PipelineError::Internal("age over 200 slipped past validation"));
  }
  Ok(age)
}

fn run(records: &[&str], fail_fast: bool) {
  for r in records {
      match process(r) {
          Ok(age) => println!("ok: {age}"),
          Err(PipelineError::Internal(msg)) => {
              println!("ABORT (bug): {msg}");   // fatal: ignores the policy
              return;
          }
          Err(PipelineError::Data(e)) => {
              // TODO: fail_fast => abort; otherwise => quarantine to DLQ and keep going
              todo!("decide this record's fate from the policy")
          }
      }
  }
}

fn main() {
  let records = ["42", "oops", "37"];
  run(&records, false); // Continue: 'oops' should quarantine, '37' still runs
  run(&records, true);  // FailFast: 'oops' should abort
}

#[derive(Debug)] enum DataError { BadField(String) } #[derive(Debug)] enum PipelineError { Data(DataError), Internal(&'static str), } impl From<DataError> for PipelineError { fn from(e: DataError) -> Self { PipelineError::Data(e) } } fn process(s: &str) -> Result<u32, PipelineError> { let age: u32 = s.parse().map_err(|_| DataError::BadField(format!("not a number: {s}")))?; if age > 200 { return Err(PipelineError::Internal("age over 200 slipped past validation")); } Ok(age) } fn run(records: &[&str], fail_fast: bool) { for r in records { match process(r) { Ok(age) => println!("ok: {age}"), Err(PipelineError::Internal(msg)) => { println!("ABORT (bug): {msg}"); // fatal: ignores the policy return; } Err(PipelineError::Data(e)) => { // TODO: fail_fast => abort; otherwise => quarantine to DLQ and keep going todo!("decide this record's fate from the policy") } } } } fn main() { let records = ["42", "oops", "37"]; run(&records, false); // Continue: 'oops' should quarantine, '37' still runs run(&records, true); // FailFast: 'oops' should abort }

💡 Hint 1

Only the recoverable class consults the policy. if fail_fast { … return; } aborts; the else/fall-through path prints a “quarantine to DLQ” line and lets the loop continue to the next record. The Internal arm above does NOT branch on fail_fast; that’s the whole point.

Show solution

Err(PipelineError::Data(e)) => {
    if fail_fast {
        println!("ABORT (fail-fast): {e:?}");
        return;
    }
    println!("quarantine to DLQ, keep going: {e:?}");
}

The asymmetry is now visible in the code shape: the Data arm reads fail_fast and may continue; the Internal arm never looks at it and always returns. Fate follows kind, and the policy only ever governs the recoverable class. This mirrors dispatch_transform_eval_error, which is the only place that consults ErrorStrategy, and which only ever sees recoverable eval errors.

Faded: write the From-powered propagation

Your turn with much less scaffolding. Two leaf errors come from two subsystems. Write the two From impls and the validate function so that ? lifts each leaf into PipelineError automatically, with no match and no map_err. The rules: a parse failure is a recoverable Data error; a negative count is the impossible case, so construct Internal directly (don’t route it through a fallible call).

rust // editable

reset ▶ run

#[derive(Debug)]
struct ParseError(String);     // leaf error from the "format" layer
#[derive(Debug)]
struct RangeError(String);     // leaf error from the "coercion" layer

#[derive(Debug)]
enum PipelineError {
  Data(ParseError),
  Coercion(RangeError),
  Internal(&'static str),    // a bug — always fatal, constructed directly
}

// TODO: write From<ParseError> and From<RangeError> for PipelineError
// so that `?` can lift each leaf into the right variant.

fn parse_count(s: &str) -> Result<i64, ParseError> {
  s.parse().map_err(|_| ParseError(format!("not an integer: {s}")))
}

fn nonneg(n: i64) -> Result<u64, RangeError> {
  if n < 0 { Err(RangeError(format!("negative: {n}"))) } else { Ok(n as u64) }
}

// TODO: return Result<u64, PipelineError>. Use `?` on parse_count and nonneg
// (their From impls fire). If the result exceeds 1_000_000, that's 'impossible'
// data that validation should have caught — return Internal directly.
fn validate(s: &str) -> Result<u64, PipelineError> {
  // your code here
  todo!()
}

fn main() {
  for s in ["500000", "oops", "-4", "9999999"] {
      match validate(s) {
          Ok(n) => println!("ok: {n}"),
          Err(PipelineError::Internal(m)) => println!("ABORT (bug): {m}"),
          Err(e) => println!("recoverable, would DLQ: {e:?}"),
      }
  }
}

#[derive(Debug)] struct ParseError(String); // leaf error from the "format" layer #[derive(Debug)] struct RangeError(String); // leaf error from the "coercion" layer #[derive(Debug)] enum PipelineError { Data(ParseError), Coercion(RangeError), Internal(&'static str), // a bug — always fatal, constructed directly } // TODO: write From<ParseError> and From<RangeError> for PipelineError // so that `?` can lift each leaf into the right variant. fn parse_count(s: &str) -> Result<i64, ParseError> { s.parse().map_err(|_| ParseError(format!("not an integer: {s}"))) } fn nonneg(n: i64) -> Result<u64, RangeError> { if n < 0 { Err(RangeError(format!("negative: {n}"))) } else { Ok(n as u64) } } // TODO: return Result<u64, PipelineError>. Use `?` on parse_count and nonneg // (their From impls fire). If the result exceeds 1_000_000, that's 'impossible' // data that validation should have caught — return Internal directly. fn validate(s: &str) -> Result<u64, PipelineError> { // your code here todo!() } fn main() { for s in ["500000", "oops", "-4", "9999999"] { match validate(s) { Ok(n) => println!("ok: {n}"), Err(PipelineError::Internal(m)) => println!("ABORT (bug): {m}"), Err(e) => println!("recoverable, would DLQ: {e:?}"), } } }

Show solution

impl From<ParseError> for PipelineError {
    fn from(e: ParseError) -> Self { PipelineError::Data(e) }
}
impl From<RangeError> for PipelineError {
    fn from(e: RangeError) -> Self { PipelineError::Coercion(e) }
}

fn validate(s: &str) -> Result<u64, PipelineError> {
    let raw = parse_count(s)?;     // From<ParseError> fires on Err
    let count = nonneg(raw)?;      // From<RangeError> fires on Err
    if count > 1_000_000 {
        // an invariant validation was supposed to enforce — a bug, not bad data
        return Err(PipelineError::Internal("count over 1_000_000 slipped past validation"));
    }
    Ok(count)
}

Two From impls, and now both ? calls lift their foreign error into PipelineError with no ceremony, exactly how the real error.rs absorbs six subsystem errors. The Internal case is constructed directly and returned, never produced by a fallible subsystem call, so it can never be mistaken for a recoverable error or routed to the DLQ. That’s the propagation seam and the taxonomy split, both produced by your own code.

Retrieval checkpoint

✓ Checkpoint — error handling

💡 Hint 1

Count the impl From< lines; each one is a subsystem ? can absorb. Then read a couple of the variant doc comments that contain “Always aborts the run”; those are the fatal class.

What the tour establishes

PipelineError is hand-written with six From impls, so ? lifts ConfigError, FormatError, EvalError, SchemaError, io::Error, and SpillError into it. The “Always aborts the run” grep lists the fatal variants (Internal, MemoryBudgetExceeded, schema/sort-order violations, …). dispatch_transform_eval_error is the only router that consults ErrorStrategy, and it only ever sees recoverable per-record eval errors; the fatal variants never reach it. Fate follows kind.

// quick check

What does writing `let r = some_subsystem_call()?;` inside a fn returning Result<_, PipelineError> rely on?

• A catch-all _ => arm somewhere
• A From<SubsystemError> for PipelineError impl, which ? uses to convert the error before returning it
• The thiserror derive macro

// quick check

Why does an Internal error abort the run even under ErrorStrategy::Continue, while an Eval error can be sent to the DLQ?

• Internal errors are slower to recover from
• Internal means a Clinker bug, a violated invariant, so quarantining it would hide a broken engine behind a 'bad record' label; only genuine per-record data errors are DLQ-eligible
• Eval errors are always more severe than Internal errors

// quick check

Two engineers describe ErrorStrategy::Continue. Which statement is the MISCONCEPTION to correct?

• It routes recoverable per-record errors to the DLQ and keeps processing the rest of the rows
• It means the run continues through every error, so even an Internal (clinker-bug) error is swallowed and the job finishes
• It is a policy about your data's messiness, not a property of any one error variant

// quick check

Spaced recall from the Option-and-Result lesson: in `fn sum(a: &str, b: &str) -> Result<i64, MyErr> { let x = coerce(a)?; let y = coerce(b)?; Ok(x + y) }`, what does the `?` after `coerce(a)` do when coerce returns Err?

• It panics and unwinds the stack
• It returns that Err from `sum` immediately (converting the error type via From), skipping the rest of the function
• It loops back and retries coerce(a) until it succeeds

You should be able to:

• You can name the subsystem each PipelineError variant aggregates (Config, Format, Eval, Io, …)
• You can explain how a From<E> impl lets ? convert a subsystem error into PipelineError
• You can name a recoverable variant and a fatal variant and say what each does under Continue
• You can explain why Internal is deliberately not DLQ-eligible, and that dispatch_transform_eval_error is the only router that reads ErrorStrategy

Verify in the checkout:

grep -n 'pub enum PipelineError' crates/clinker-plan/src/error.rs
grep -n 'impl From<' crates/clinker-plan/src/error.rs
grep -n 'Always aborts the run' crates/clinker-plan/src/error.rs | head
grep -n 'fn dispatch_transform_eval_error' crates/clinker-exec/src/executor/dispatch.rs

Connections

You can now read the engine’s failure taxonomy and say why some errors quarantine a record while others kill the run. You also saw that PipelineError is hand-written. The next lesson shows exactly what that takes, and the one after it meets thiserror, the derive that generates the same impls. CXL, the expression language that has shown up from the outside all module, gets the deep lesson that closes Planning & Expressions.

Prerequisites

• The plan/runtime boundary

→

Leads to

• Building an error type by hand

Go deeper (optional, one-directional):

• Want to design an error type like PipelineError by hand: Display, source, and From? Skip ahead →
• Want Result and the ? operator drilled on a standalone parser? Read the docs →

Glossary terms used: error strategy, DLQ, Result / From / ?.

Mark this lesson complete