Missing cells & fallible parses — Option and Result

Clinker’s streaming reader answers two questions on every record: is this field even here? and did reading it fail? It encodes both in the type its hot loop returns, Result<Option<Record>, FormatError>, so a caller can’t read a row without first confronting “maybe absent” and “maybe failed.” You already met absence in passing while moving and borrowing a record, where a field lookup returned Option<&Value>: “maybe there, maybe not.” This lesson is the deep pass: how the engine stacks absence inside failure into one composed type, and reads it as three distinct outcomes. The Rust tools underneath are Option and Result; we use just enough of each to read the real signature, and The Rust Book, ch. 9 is the canonical treatment of Result and the ? operator if you want them from first principles.

Prerequisites

• Who owns a record

→

Leads to

• Provenance

What you’ll be able to do

• Read Option<T> and Result<T, E> in real engine signatures and name what each variant encodes.
• Decompose the reader type Result<Option<Record>, FormatError> into its three outcomes and explain what each one means for a streaming job.
• Write a fallible coercion function that returns the composed type, distinguishing a clean absence from a real error.
• Distinguish a recoverable (dead-letter) error from a fatal one, and say why the type system makes the split visible.

New terms in this lesson (each is also expanded inline at the point you first need it):

• Option<T>: absence encoded in the type.
• Result<T, E>: failure encoded in the type.
• Result<Option<T>, E>: the two composed, read as three outcomes.
• the ? operator: one-character early-return-on-error, pre-taught where it appears below.

Predict first

✋ Predict first

A streaming reader has this signature: fn next_record(&mut self) -> Result<Option<Record>, FormatError>. It returns a Result wrapping an Option. How many distinct things can the caller learn from one call, and what are they? Write down your count and your labels before revealing.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

There are three outcomes, not two and not four, and reading them is the whole skill this lesson builds:

• Ok(Some(record)) is a value: here is the next row, process it.
• Ok(None) is clean end-of-input: the file is exhausted, stop the loop. No error.
• Err(e) is something went wrong: a malformed line, a broken encoding.

The trap is collapsing Ok(None) and Err(e) into one “nothing came back” case. They are opposites: one is the expected, successful end of a finite source; the other is a failure. The type keeps them apart so your loop can stop cleanly on one and react to the other. We’ll build up to reading this fluently, but notice you predicted it before you saw a line of the type’s internals. Hold that count of three; everything below earns it.

Start concrete: one Option, one Result

Before composing anything, look at each half on a real value.

Option<T>: maybe there, maybe not

You already met one in the last lesson: resolve(&self, name: &str) -> Option<&Value>. An Option<T> is honest about a fact the type system would otherwise hide: a field you ask for might not exist. It’s Some(value) when it’s there and None when it isn’t. There’s no null to forget to check; to get at the value you must handle the None arm.

Here it is on a concrete field lookup you can run:

rust // editable

reset ▶ run

#[derive(Debug)]
enum Value { Integer(i64), Text(String) }

// A field lookup either finds a value or it doesn't.
// `Option` puts that "maybe absent" fact into the return type.
fn lookup<'a>(fields: &'a [(&str, Value)], name: &str) -> Option<&'a Value> {
  for (key, value) in fields {
      if *key == name {
          return Some(value); // found it: a borrow of the value
      }
  }
  None // not present — and the caller can't forget to handle this
}

fn main() {
  let row = [("id", Value::Integer(7)), ("status", Value::Text(String::from("active")))];

  match lookup(&row, "status") {
      Some(v) => println!("found: {v:?}"),
      None => println!("absent"),
  }
  match lookup(&row, "missing") {
      Some(v) => println!("found: {v:?}"),
      None => println!("absent"),
  }
}

#[derive(Debug)] enum Value { Integer(i64), Text(String) } // A field lookup either finds a value or it doesn't. // `Option` puts that "maybe absent" fact into the return type. fn lookup<'a>(fields: &'a [(&str, Value)], name: &str) -> Option<&'a Value> { for (key, value) in fields { if *key == name { return Some(value); // found it: a borrow of the value } } None // not present — and the caller can't forget to handle this } fn main() { let row = [("id", Value::Integer(7)), ("status", Value::Text(String::from("active")))]; match lookup(&row, "status") { Some(v) => println!("found: {v:?}"), None => println!("absent"), } match lookup(&row, "missing") { Some(v) => println!("found: {v:?}"), None => println!("absent"), } }

Notice the return type is Option<&Value>, not Option<Value>. That & is the borrow you studied last lesson, lending a window onto the value already in the row instead of copying it out. The Option wraps that borrow; it doesn’t change why it’s a borrow.

Result<T, E>: maybe it failed

Reading a value is one thing; transforming one can fail. Coercing the text "active" to an integer is nonsense. A Result<T, E> makes that failure a value you handle rather than an exception that unwinds the stack: it’s Ok(value) on success and Err(error) on failure.

rust // editable

reset ▶ run

#[derive(Debug)]
enum Value { Integer(i64), Text(String) }

// Coercing a text cell to an integer can FAIL. That's not absence —
// it's an error. `Result` puts "maybe failed" into the return type.
fn to_int(v: &Value) -> Result<i64, String> {
  match v {
      Value::Integer(n) => Ok(*n),
      Value::Text(s) => s
          .parse::<i64>()
          .map_err(|_| format!("not an integer: {s:?}")),
  }
}

fn main() {
  let good = Value::Text(String::from("15200"));
  let bad = Value::Text(String::from("active"));

  match to_int(&good) {
      Ok(n) => println!("ok: {n}"),
      Err(e) => println!("failed: {e}"),
  }
  match to_int(&bad) {
      Ok(n) => println!("ok: {n}"),
      Err(e) => println!("failed: {e}"),
  }
}

#[derive(Debug)] enum Value { Integer(i64), Text(String) } // Coercing a text cell to an integer can FAIL. That's not absence — // it's an error. `Result` puts "maybe failed" into the return type. fn to_int(v: &Value) -> Result<i64, String> { match v { Value::Integer(n) => Ok(*n), Value::Text(s) => s .parse::<i64>() .map_err(|_| format!("not an integer: {s:?}")), } } fn main() { let good = Value::Text(String::from("15200")); let bad = Value::Text(String::from("active")); match to_int(&good) { Ok(n) => println!("ok: {n}"), Err(e) => println!("failed: {e}"), } match to_int(&bad) { Ok(n) => println!("ok: {n}"), Err(e) => println!("failed: {e}"), } }

Two halves, each on a concrete value: Option says present or absent; Result says succeeded or failed. Keep them separate in your head. That separation is exactly what makes the composed type readable.

Compose them: the streaming-record read pattern

Now put the halves together. A reader streams records one at a time, and on each call two independent things can be true:

1. Is there a next record, or has the finite source run out? → that’s an Option: Some(record) or None.
2. Did the attempt to read it succeed, or did it hit a malformed line? → that’s a Result: Ok(...) or Err(e).

Stack the Option inside the Result and you get the type the engine actually uses:

Result<Option<Record>, FormatError>
//     ^^^^^^^^^^^^^^^                 inner: is there a next record? (Option)
// ^^^^^^                              outer: did the read succeed?   (Result)

Read it from the outside in, and it resolves to the three outcomes you predicted at the top:

// Ok(Some(record)) → a value           (here is the next row)
// Ok(None)         → clean end-of-input (the finite source is exhausted; stop)
// Err(e)           → something went wrong (a malformed line, a broken encoding)

Ok(None) and Err(e) are the pair people conflate. They are opposites: end-of-input is a successful outcome that happens to carry no record; an error is a failure. The composed type forces your loop to treat them differently, and that’s the point.

Worked → completion → faded

Now you read and write the composed type, scaffolded down to your own.

Worked: read all three outcomes

Here is a self-contained reader that returns the engine’s exact type and a loop that handles every outcome. Run it and watch the three branches fire.

rust // editable

reset ▶ run

#[derive(Debug)]
struct Record { id: i64 }

#[derive(Debug)]
struct FormatError(String);

// A streaming reader returns the SAME composed type the engine uses:
// Result<Option<Record>, FormatError>. Read it as three outcomes.
fn next_record(remaining: &mut Vec<Result<Record, FormatError>>) -> Result<Option<Record>, FormatError> {
  match remaining.pop() {
      Some(Ok(rec)) => Ok(Some(rec)),  // a row
      Some(Err(e)) => Err(e),          // something went wrong
      None => Ok(None),                // clean end of input
  }
}

fn main() {
  // Built back-to-front because pop() takes from the end.
  let mut source = vec![
      Err(FormatError(String::from("malformed line 3"))),
      Ok(Record { id: 2 }),
      Ok(Record { id: 1 }),
  ];

  loop {
      match next_record(&mut source) {
          Ok(Some(rec)) => println!("row: {rec:?}"),
          Ok(None) => {
              println!("end of input (clean)");
              break;
          }
          Err(e) => println!("error, would DLQ or abort: {e:?}"),
      }
  }
}

#[derive(Debug)] struct Record { id: i64 } #[derive(Debug)] struct FormatError(String); // A streaming reader returns the SAME composed type the engine uses: // Result<Option<Record>, FormatError>. Read it as three outcomes. fn next_record(remaining: &mut Vec<Result<Record, FormatError>>) -> Result<Option<Record>, FormatError> { match remaining.pop() { Some(Ok(rec)) => Ok(Some(rec)), // a row Some(Err(e)) => Err(e), // something went wrong None => Ok(None), // clean end of input } } fn main() { // Built back-to-front because pop() takes from the end. let mut source = vec![ Err(FormatError(String::from("malformed line 3"))), Ok(Record { id: 2 }), Ok(Record { id: 1 }), ]; loop { match next_record(&mut source) { Ok(Some(rec)) => println!("row: {rec:?}"), Ok(None) => { println!("end of input (clean)"); break; } Err(e) => println!("error, would DLQ or abort: {e:?}"), } } }

Completion: write the third arm

Below, two match arms are written for you. The Err(_) arm is the one to complete: a read error means this record can’t be produced. Under a continue strategy the engine routes it to the dead-letter queue and keeps going; otherwise it aborts. Fill in a string that captures that.

rust // editable

reset ▶ run

#[derive(Debug)]
struct Record { id: i64 }

#[derive(Debug)]
struct FormatError(String);

fn classify(outcome: Result<Option<Record>, FormatError>) -> &'static str {
  match outcome {
      Ok(Some(_)) => "a row to process",
      Ok(None) => "clean end of input — stop the loop",
      Err(_) => "TODO: what does an Err mean for the loop?", // <- complete this arm
  }
}

fn main() {
  println!("{}", classify(Ok(Some(Record { id: 1 }))));
  println!("{}", classify(Ok(None)));
  println!("{}", classify(Err(FormatError(String::from("bad byte")))));
}

#[derive(Debug)] struct Record { id: i64 } #[derive(Debug)] struct FormatError(String); fn classify(outcome: Result<Option<Record>, FormatError>) -> &'static str { match outcome { Ok(Some(_)) => "a row to process", Ok(None) => "clean end of input — stop the loop", Err(_) => "TODO: what does an Err mean for the loop?", // <- complete this arm } } fn main() { println!("{}", classify(Ok(Some(Record { id: 1 })))); println!("{}", classify(Ok(None))); println!("{}", classify(Err(FormatError(String::from("bad byte"))))); }

💡 Hint 1

An Err is neither a row nor a clean stop. It’s the only outcome where something failed, so the loop must decide between recover (DLQ) and abort.

Show solution

Err(_) => "an error — DLQ the row or abort the run",

The exhaustive match is what guarantees you considered all three. Drop any arm and the compiler refuses to build; that’s the exhaustiveness you met back in reading a value with match, now doing real work on a composed type.

Faded: write the whole function

Your turn with much less scaffolding. Write coerce_to_int returning Result<Option<i64>, CoercionError>. The rule: a Null cell is legitimately absent (map it to Ok(None), since absence is not failure); an Integer is Ok(Some(n)); text that parses is Ok(Some(n)); text that doesn’t parse is the only Err.

rust // editable

reset ▶ run

#[derive(Debug)]
enum Value { Null, Integer(i64), Text(String) }

#[derive(Debug)]
enum CoercionError {
  ParseFailure { input: String, target: &'static str },
}

// TODO: return Ok(None) for Null, Ok(Some(n)) for a number or parseable text,
// and Err(CoercionError::ParseFailure { .. }) when text won't parse.
fn coerce_to_int(v: &Value) -> Result<Option<i64>, CoercionError> {
  // your code here
  todo!()
}

fn main() {
  let cells = [
      Value::Integer(15200),
      Value::Text(String::from("42")),
      Value::Null,
      Value::Text(String::from("active")),
  ];
  for cell in &cells {
      match coerce_to_int(cell) {
          Ok(Some(n)) => println!("value: {n}"),
          Ok(None) => println!("null — left as absent"),
          Err(e) => println!("would DLQ: {e:?}"),
      }
  }
}

#[derive(Debug)] enum Value { Null, Integer(i64), Text(String) } #[derive(Debug)] enum CoercionError { ParseFailure { input: String, target: &'static str }, } // TODO: return Ok(None) for Null, Ok(Some(n)) for a number or parseable text, // and Err(CoercionError::ParseFailure { .. }) when text won't parse. fn coerce_to_int(v: &Value) -> Result<Option<i64>, CoercionError> { // your code here todo!() } fn main() { let cells = [ Value::Integer(15200), Value::Text(String::from("42")), Value::Null, Value::Text(String::from("active")), ]; for cell in &cells { match coerce_to_int(cell) { Ok(Some(n)) => println!("value: {n}"), Ok(None) => println!("null — left as absent"), Err(e) => println!("would DLQ: {e:?}"), } } }

Show solution

fn coerce_to_int(v: &Value) -> Result<Option<i64>, CoercionError> {
    match v {
        Value::Null => Ok(None),            // absent: clean, not an error
        Value::Integer(n) => Ok(Some(*n)),
        Value::Text(s) => match s.parse::<i64>() {
            Ok(n) => Ok(Some(n)),
            Err(_) => Err(CoercionError::ParseFailure {
                input: s.clone(),
                target: "Integer",
            }),
        },
    }
}

This is the three-outcome type again, now produced by your code: Ok(None) for clean absence, Ok(Some(n)) for a value, Err(e) for a real failure. That .clone() on s is a deliberate, owned copy for the error message, the kind of clone the last lesson called the right call, because the error needs to outlive the borrowed cell.

Real-source grounding

Here is the trait method the worked example mirrors. The engine’s streaming reader returns the composed type on every call:

clinker-format ·traits.rs ·next_record fn @19acdcb4

fn next_record(&mut self) -> Result<Option<Record>, FormatError>;
//                           Ok(Some(record)) → a row
//                           Ok(None)         → end of input (clean)
//                           Err(e)           → something went wrong

What the compiler enforces: a caller can’t read a Record out of this without first peeling the Result (handling Err) and then the Option (handling None). There is no path to the inner Record that skips either check.

What a junior might misread: treating Ok(None) as an error condition (so a clean, finite file looks like a failure), or treating Err(e) as “no more data” (so a malformed line silently ends the job). The type makes both mistakes loud, but only if you read all three arms.

And here is the typed error a coercion produces, the one your faded task modeled:

clinker-record ·coercion.rs ·CoercionError type @19acdcb4

pub enum CoercionError {
    TypeMismatch { from: &'static str, to: &'static str, value: String },
    ParseFailure { input: String, target: &'static str },
}

The real coerce_to_int returns Result<Value, CoercionError>, and there’s a lenient companion that returns an Option instead by throwing the error away:

clinker-record ·coercion.rs ·coerce_to_int fn @19acdcb4

// strict: the failure is a typed value you must handle
pub fn coerce_to_int(value: &Value) -> Result<Value, CoercionError> { /* ... */ }

// lenient: same logic, but the error is discarded into None
pub fn coerce_to_int_lenient(value: &Value) -> Option<Value> {
    coerce_to_int(value).ok() // .ok() turns Ok(v) -> Some(v), Err(_) -> None
}

That .ok() is the clearest possible statement of the difference between the two types: a Result carries why it failed, where an Option only records that something is absent. The strict path keeps the reason so the engine can put it in a dead-letter entry; the lenient path is for callers that genuinely don’t care which way it went.

The ? operator: propagating errors without a staircase

Once functions return Result, you chain fallible steps constantly. Writing a full match at every step would bury the logic.

🌱 New here? — the ? operator

The ? operator is one-character error propagation. Written after a Result-returning call, let x = something()?; means: if the call is Err, return that Err from the current function right here; if it’s Ok, unwrap the value and keep going. The current function’s return type must itself be a Result (so there’s an Err to return into), and ? converts the error type along the way via the From trait. It’s the same early-return you’d write by hand, compressed to a single character.

rust // editable

reset ▶ run

#[derive(Debug)]
struct CoercionError(String);

fn coerce_to_int(s: &str) -> Result<i64, CoercionError> {
  s.parse::<i64>().map_err(|_| CoercionError(format!("not an integer: {s:?}")))
}

// `?`: on Err, return that Err from sum_two right here; on Ok, unwrap and continue.
fn sum_two(a: &str, b: &str) -> Result<i64, CoercionError> {
  let x = coerce_to_int(a)?;
  let y = coerce_to_int(b)?;
  Ok(x + y)
}

fn main() {
  println!("{:?}", sum_two("20", "22"));     // Ok(42)
  println!("{:?}", sum_two("20", "active")); // Err(CoercionError("not an integer: \"active\""))
}

#[derive(Debug)] struct CoercionError(String); fn coerce_to_int(s: &str) -> Result<i64, CoercionError> { s.parse::<i64>().map_err(|_| CoercionError(format!("not an integer: {s:?}"))) } // `?`: on Err, return that Err from sum_two right here; on Ok, unwrap and continue. fn sum_two(a: &str, b: &str) -> Result<i64, CoercionError> { let x = coerce_to_int(a)?; let y = coerce_to_int(b)?; Ok(x + y) } fn main() { println!("{:?}", sum_two("20", "22")); // Ok(42) println!("{:?}", sum_two("20", "active")); // Err(CoercionError("not an integer: \"active\"")) }

The engine’s fallible paths are threaded with ? end to end. You’ll meet the full error vocabulary, how Clinker defines one error type that every step’s ? converts into, in the error-handling lesson; this lesson only needs you to read ? where it appears.

Already fluent in Result and the ? operator from another language? Read the docs →

Why-bridge: recoverable vs fatal, and the DLQ

Why does the engine bother splitting absence from failure so carefully? Because not every error should kill a job. If one row in a million can’t be coerced ("active".to_int() is nonsense), Clinker can route that single bad record to the dead-letter queue (the dlq count you saw in --dry-run) and keep going, instead of aborting the whole run.

That decision rides on the type split you just learned:

• A data error on one row (a CoercionError, or a malformed line surfaced as Err(FormatError)) is recoverable. It can be dead-lettered; the run continues.
• An invariant error, a bug or an impossible state, is fatal. The run aborts.

The strict coerce_to_int returning Result (not the lenient Option) is what makes the recoverable case carry its reason all the way to the dead-letter entry, so an operator sees “row 42: failed to parse ‘active’ as Integer” rather than a silent drop.

Apply: predict, then repair

This is a generation task, not recognition. Read the function below. It’s meant to be broken: it tries to use an Option as if it were the value inside it.

// WON'T COMPILE — kept here as a deliberate counterexample.
fn lookup_id(present: bool) -> Option<i64> {
    if present { Some(7) } else { None }
}

fn main() {
    let found = lookup_id(true);
    let next = found + 1; // <- error here
    println!("{next}");
}

Predict the compiler first. What error does let next = found + 1; produce, and why?

Show the compiler's answer

error[E0369]: cannot add `{integer}` to `Option<i64>`
  --> src/main.rs
   |
   |     let next = found + 1;
   |                ----- ^ - {integer}
   |                |
   |                Option<i64>

found is an Option<i64>, not an i64. You can’t do arithmetic on “maybe a number”; you must first handle the None case to get at the Some value. The fix makes the absence explicit:

let next = match lookup_id(true) {
    Some(n) => n + 1,
    None => 0, // or return early, or propagate — but you must DECIDE
};

This is the entire safety promise of Option in one error message: the compiler will not let you pretend an absent value is present. The same is true one layer up. You can’t read a Record out of Result<Option<Record>, _> without peeling both wrappers.

Retrieval checkpoint

✓ Checkpoint

// quick check

A reader's next_record returns Result<Option<Record>, FormatError>. What does Ok(None) mean?

• An error occurred while reading
• A clean end of input: the finite source is exhausted
• An empty record with no fields

// quick check

In the last lesson, FieldResolver::resolve returned Option<&Value> instead of Option<Value>. Why the & inside the Option?

• To make the Option smaller
• So reading a field borrows the existing value instead of copying it out, and copies can allocate
• Because Option can only hold references

// quick check

Two engineers describe coerce_to_int_lenient (which returns Option). Which statement is the MISCONCEPTION to correct?

• It discards the error reason: fine for callers that don't need it, wrong when the engine needs the reason for a DLQ entry
• Its Option<Value> result is interchangeable with the strict Result<Value, CoercionError>, so you can swap one for the other anywhere
• It is built from the strict version by calling .ok() on its Result

You should be able to:

• You can name the three outcomes of Result<Option<Record>, FormatError> and what each means for a loop
• You can explain why Ok(None) and Err(e) are opposites, not the same 'nothing' case
• You wrote a function returning Result<Option<i64>, CoercionError> that maps Null to Ok(None), a value to Ok(Some(n)), and unparseable text to Err
• You can say what .ok() does and when the lenient Option path is the wrong choice (when the engine needs the reason for a DLQ entry)
• You predicted the 'cannot add to Option<i64>' error and gave a fix that handles None

Verify in the checkout:

cargo build -p clinker-record --offline
cargo test -p clinker-record --offline test_coerce_string_to_int_invalid
cargo doc -p clinker-record --no-deps --offline

Connections

You can now read absence and failure in the types, decompose the composed reader type into its three outcomes, and write a function that produces it. Next: the small struct that rides along with every record so the engine always knows where a row came from.

Prerequisites

• Who owns a record

→

Leads to

• Provenance

Go deeper on the Rust (optional, one-directional, for the same concepts taught from first principles in The Rust Book):

• Want Option from first principles, with no engine framing? Read the docs →
• Want Result and the ? operator drilled on a standalone parser? Read the docs →

Glossary terms used: Option, Result, Result<Option<T>>, borrow.

Mark this lesson complete