Add a reader/writer/format

In the IO seam you met formats from the outside: they are an open, runtime-chosen plug-in surface, so the engine reaches them through Box<dyn FormatReader>. Now you’re on the inside, adding one. This lesson shows the full change-set, and it pulls together two earlier ideas: the dyn seam is where your code plugs in, and the closed exhaustive match from dispatching a node is what forces every dispatch site to acknowledge your new format.

Prerequisites

• Add a CXL builtin

→

Leads to

• Add a transform/operator

What you’ll be able to do

• Name the two required trait methods on FormatReader and on FormatWriter, and what Ok(None) from next_record signals.
• Implement the FormatReader trait for a new type so it plugs into the engine’s boxed-reader seam.
• Trace the two-stage path from a YAML type: string to a concrete reader: serde maps the string to an enum variant, then an exhaustive match maps the variant to a Box<dyn FormatReader>.
• List the edit sites a new format touches and distinguish which one the compiler makes mandatory from the ones it does not.

New terms in this lesson (each is also expanded inline at the point you first need it):

• format reader/writer trait: FormatReader/FormatWriter are the two contracts a format satisfies.
• the seam: the plug-in boundary those traits create, expressed at the boundary as a trait object (Box<dyn FormatReader>).
• dyn: dynamic dispatch, so one boxed reader can be any format.
• format registration: the act of wiring a new variant into the config enum and its dispatch match; pre-taught where it appears below.

Predict first

✋ Predict first

Reader dispatch is one function: fn build_format_reader(...) -> Box<dyn FormatReader>, a match over the InputFormat enum with NO catch-all _ => arm. You add InputFormat::Tsv but forget to touch build_format_reader. Does the crate compile, fail to compile, or compile and pick a wrong arm at run time? Write down your answer and one sentence of why before revealing.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

It fails to compile, and that is the whole safety property this lesson leans on.

build_format_reader matches InputFormat with no _ => wildcard, so the match is exhaustive: it must name every variant. The moment you add Tsv, the match no longer covers all variants, and rustc reports a non-exhaustive-match error that points at the exact function and the missing variant. You cannot ship a half-wired format by accident.

The trap is assuming a forgotten arm is a run-time bug: a wrong reader chosen, or a panic when a TSV job runs. It is caught at compile time instead, before any job runs, because the enum is closed and the match has no catch-all. That is the closed-enum guarantee from dispatching a node doing your change-management for you. The open Box<dyn FormatReader> return type lets your reader plug in; the exhaustive match hands you the list of sites to wire. Hold that distinction, compiler-mandatory versus merely-necessary; everything below sorts the edit sites by it.

Two traits, two required methods each

A format is whatever can stream Records in and out. The two contracts are tiny:

clinker-format ·traits.rs ·FormatReader trait @19acdcb4

/// Streaming record reader. Yields records one at a time.
///
/// `&mut self` on `schema()` because some formats (e.g. CSV) must read
/// the first row to discover column names. Must be `Send` for executor
/// ownership transfer; not `Sync` — single-threaded streaming.
pub trait FormatReader: Send {
    fn schema(&mut self) -> Result<Arc<Schema>, FormatError>;
    fn next_record(&mut self) -> Result<Option<Record>, FormatError>;
    // ...plus a few defaulted methods for envelopes / multi-file readers
}

clinker-format ·traits.rs ·FormatWriter trait @19acdcb4

/// Streaming record writer. Consumes records one at a time.
pub trait FormatWriter: Send {
    fn write_record(&mut self, record: &Record) -> Result<(), FormatError>;
    fn flush(&mut self) -> Result<(), FormatError>;
    // ...plus defaulted begin_document / end_document for envelope framing
}

Two required methods on each, both fallible, both streaming. next_record returning Ok(None) means clean end-of-stream, not an error, the same three-outcome read you learned in Option and Result. The : Send bound is the same one the worker pool explained: the executor moves readers onto worker threads, so a reader must be Send. A minimal new format implements just those two-each methods; the defaulted ones (envelopes, multi-file) are opt-in.

A concrete reader: CSV

Here’s the shape of a real implementation: CsvReader, the simplest complete one:

clinker-format ·reader.rs ·CsvReader type @19acdcb4

pub struct CsvReader<R: Read> {
    inner: csv::Reader<SkipBom<R>>,
    schema: Option<Arc<Schema>>,    // discovered from the header, cached
    config: CsvReaderConfig,
    row_count: u64,
    record_buf: csv::StringRecord,
}

impl<R: Read + Send> FormatReader for CsvReader<R> {
    fn schema(&mut self) -> Result<Arc<Schema>, FormatError> {
        self.ensure_schema()                  // reads the header row once
    }
    fn next_record(&mut self) -> Result<Option<Record>, FormatError> {
        let schema = self.ensure_schema()?;
        if !self.inner.read_record(&mut self.record_buf)? {
            return Ok(None);                  // end of stream
        }
        let values = self.record_buf.iter().map(|f| Value::String(f.into())).collect();
        Ok(Some(Record::new(schema, values)))
    }
}

That’s the whole pattern: hold the underlying byte source plus a cached Arc<Schema>, and have next_record pull one row and emit Ok(Some(Record)) until the source is dry. The impl is generic over R: Read + Send, not tied to files.

From a YAML string to your reader, in two stages

A pipeline says type: csv in YAML. How does that string reach CsvReader? Not by a string-match you write; it’s a two-stage hop, and tracing it tells you exactly what to edit.

Stage 1, string to enum variant, by serde. The config layer has a closed enum of known formats, adjacently tagged so serde maps type: csv to a variant automatically:

clinker-plan ·format.rs ·InputFormat type @19acdcb4

#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type", content = "options", rename_all = "snake_case")]
pub enum InputFormat {
    Csv(Option<CsvInputOptions>),
    Json(Option<JsonInputOptions>),
    Xml(Option<XmlInputOptions>),
    FixedWidth(Option<FixedWidthInputOptions>),
    Edifact(Option<EdifactInputOptions>),
    X12(Option<X12InputOptions>),
    Hl7(Option<Hl7InputOptions>),
    Swift(Option<SwiftInputOptions>),
}

The #[serde(tag = "type", rename_all = "snake_case")] is what turns the string "csv" into the variant InputFormat::Csv. You never write name-matching code.

Stage 2, enum variant to boxed reader, by an exhaustive match. In the executor, one function matches the variant and returns the boxed trait object:

clinker-exec ·ingest.rs ·build_format_reader fn @19acdcb4

fn build_format_reader(
    input: &clinker_plan::config::SourceConfig,
    source: ReopenableSource,
) -> Result<Box<dyn FormatReader>, PipelineError> {
    match &input.format {
        InputFormat::Csv(opts)  => Ok(Box::new(CsvReader::from_reader(/* ... */))),
        InputFormat::Json(opts) => Ok(Box::new(JsonReader::from_source(/* ... */)?)),
        InputFormat::Xml(opts)  => { /* ... */ }
        // ...one arm per variant, NO `_ =>` catch-all
    }
}

This is the enum-dispatch payoff in action: the match is wildcard-free, so the moment you add a variant to InputFormat, this function stops compiling until you add its arm. The open dyn seam (return type Box<dyn FormatReader>) and the closed enum dispatch (match with no wildcard) cooperate: dyn lets your reader plug in; the exhaustive enum makes the compiler hand you the list of sites to wire.

Two registries, not one

Writers are dispatched the same way but by a separate function (build_format_writer, matching a separate OutputFormat enum). There’s also a narrower FormatKind enum (four variants) used elsewhere; it is not the dispatch key, so don’t wire a new format through it. Reader dispatch and writer dispatch are two independent exhaustive matches.

Worked → completion → faded

Now you implement the seam yourself, scaffolded down to a from-scratch reader. Each rung adds exactly one new idea: first see a working enum→trait-object dispatch, then fill the gap the compiler flags, then write a whole FormatReader impl on your own.

Worked: the exhaustive seam, runnable end to end

Here is the enum-match-to-trait-object pattern on its own, the same division of labor as the real build_format_reader, shrunk to fit in your head. Run it and watch one boxed Box<dyn Reader> drive two different concrete readers.

rust // editable

reset ▶ run

trait Reader {
  fn next_line(&mut self) -> Option<String>;
}

struct CsvReader { rows: Vec<String> }
impl Reader for CsvReader {
  fn next_line(&mut self) -> Option<String> { self.rows.pop() }
}

struct JsonReader { rows: Vec<String> }
impl Reader for JsonReader {
  fn next_line(&mut self) -> Option<String> { self.rows.pop() }
}

enum Format { Csv, Json }   // the closed config enum, in miniature

// The return type Box<dyn Reader> is the OPEN seam: any Reader plugs in.
// The wildcard-free match is the CLOSED dispatch: every variant named.
fn build_reader(fmt: Format) -> Box<dyn Reader> {
  match fmt {
      Format::Csv  => Box::new(CsvReader { rows: vec!["a,b".into()] }),
      Format::Json => Box::new(JsonReader { rows: vec!["{}".into()] }),
  }
}

fn main() {
  for fmt in [Format::Csv, Format::Json] {   // construct both variants
      let mut r = build_reader(fmt);         // one boxed type, two impls
      while let Some(line) = r.next_line() {
          println!("{line}");
      }
  }
}

trait Reader { fn next_line(&mut self) -> Option<String>; } struct CsvReader { rows: Vec<String> } impl Reader for CsvReader { fn next_line(&mut self) -> Option<String> { self.rows.pop() } } struct JsonReader { rows: Vec<String> } impl Reader for JsonReader { fn next_line(&mut self) -> Option<String> { self.rows.pop() } } enum Format { Csv, Json } // the closed config enum, in miniature // The return type Box<dyn Reader> is the OPEN seam: any Reader plugs in. // The wildcard-free match is the CLOSED dispatch: every variant named. fn build_reader(fmt: Format) -> Box<dyn Reader> { match fmt { Format::Csv => Box::new(CsvReader { rows: vec!["a,b".into()] }), Format::Json => Box::new(JsonReader { rows: vec!["{}".into()] }), } } fn main() { for fmt in [Format::Csv, Format::Json] { // construct both variants let mut r = build_reader(fmt); // one boxed type, two impls while let Some(line) = r.next_line() { println!("{line}"); } } }

build_reader returns Box<dyn Reader> (the open seam) and dispatches through a wildcard-free match (the closed enum). That is exactly how build_format_reader turns an InputFormat variant into a Box<dyn FormatReader>.

Completion: add the variant, let the compiler flag the gap

Below, a Tsv variant has been added to Format, but build_reader was not updated, so it no longer compiles, exactly as you predicted at the top. One arm is missing. Add it so a Tsv builds a TsvReader (already written for you), and the match is exhaustive again.

rust // editable

reset ▶ run

trait Reader {
  fn next_line(&mut self) -> Option<String>;
}

struct CsvReader { rows: Vec<String> }
impl Reader for CsvReader {
  fn next_line(&mut self) -> Option<String> { self.rows.pop() }
}

struct TsvReader { rows: Vec<String> }
impl Reader for TsvReader {
  fn next_line(&mut self) -> Option<String> { self.rows.pop() }
}

enum Format { Csv, Tsv }   // Tsv is new — the match below must grow an arm

fn build_reader(fmt: Format) -> Box<dyn Reader> {
  match fmt {
      Format::Csv => Box::new(CsvReader { rows: vec!["a,b".into()] }),
      // <- the Tsv arm is missing; the match is non-exhaustive
  }
}

fn main() {
  let mut r = build_reader(Format::Tsv);
  while let Some(line) = r.next_line() {
      println!("{line}");
  }
}

trait Reader { fn next_line(&mut self) -> Option<String>; } struct CsvReader { rows: Vec<String> } impl Reader for CsvReader { fn next_line(&mut self) -> Option<String> { self.rows.pop() } } struct TsvReader { rows: Vec<String> } impl Reader for TsvReader { fn next_line(&mut self) -> Option<String> { self.rows.pop() } } enum Format { Csv, Tsv } // Tsv is new — the match below must grow an arm fn build_reader(fmt: Format) -> Box<dyn Reader> { match fmt { Format::Csv => Box::new(CsvReader { rows: vec!["a,b".into()] }), // <- the Tsv arm is missing; the match is non-exhaustive } } fn main() { let mut r = build_reader(Format::Tsv); while let Some(line) = r.next_line() { println!("{line}"); } }

💡 Hint 1

The compiler error is non-exhaustive patterns: \Format::Tsv` not covered, pointing at the match. Mirror the Csvarm: map the new variant to a boxedTsvReader`.

Show solution

fn build_reader(fmt: Format) -> Box<dyn Reader> {
    match fmt {
        Format::Csv => Box::new(CsvReader { rows: vec!["a,b\tc".into()] }),
        Format::Tsv => Box::new(TsvReader { rows: vec!["a\tb".into()] }),
    }
}

The arm you just added is the compiler-mandatory edit, the one rustc refused to let you skip. In the real engine this same arm lives in build_format_reader; the non-exhaustive-match error is what names the exact site for you.

Faded: write a whole FormatReader impl

Now the part with no scaffolding: write the trait impl itself. Implement FormatReader for LinesReader, a from-scratch format that emits each input line as a one-field record. You write both required methods: schema returns the cached schema, and next_record pulls the next line, returning Ok(None) at end-of-input. The struct, the trait, and the loop that drives it are given.

rust // editable

reset ▶ run

use std::sync::Arc;

#[derive(Debug)]
struct Schema { columns: Vec<String> }

#[derive(Debug)]
struct Record { schema: Arc<Schema>, values: Vec<String> }

#[derive(Debug)]
struct FormatError(String);

trait FormatReader {
  fn schema(&mut self) -> Result<Arc<Schema>, FormatError>;
  fn next_record(&mut self) -> Result<Option<Record>, FormatError>;
}

struct LinesReader {
  lines: Vec<String>,   // back-to-front: pop() takes the next line
  schema: Arc<Schema>,
}

// TODO: impl FormatReader for LinesReader.
//   schema(): return Ok(self.schema.clone())  (Arc::clone is cheap)
//   next_record(): pop the next line; if None, return Ok(None);
//                  otherwise wrap it in a one-field Record and return Ok(Some(..)).

fn main() {
  let schema = Arc::new(Schema { columns: vec!["line".into()] });
  let mut reader = LinesReader {
      lines: vec!["third".into(), "second".into(), "first".into()],
      schema: schema.clone(),
  };

  // The engine drives any FormatReader through exactly this loop:
  println!("schema: {:?}", reader.schema().unwrap());
  loop {
      match reader.next_record() {
          Ok(Some(rec)) => println!("row: {:?}", rec.values),
          Ok(None) => { println!("end of input (clean)"); break; }
          Err(e) => { println!("error: {e:?}"); break; }
      }
  }
}

use std::sync::Arc; #[derive(Debug)] struct Schema { columns: Vec<String> } #[derive(Debug)] struct Record { schema: Arc<Schema>, values: Vec<String> } #[derive(Debug)] struct FormatError(String); trait FormatReader { fn schema(&mut self) -> Result<Arc<Schema>, FormatError>; fn next_record(&mut self) -> Result<Option<Record>, FormatError>; } struct LinesReader { lines: Vec<String>, // back-to-front: pop() takes the next line schema: Arc<Schema>, } // TODO: impl FormatReader for LinesReader. // schema(): return Ok(self.schema.clone()) (Arc::clone is cheap) // next_record(): pop the next line; if None, return Ok(None); // otherwise wrap it in a one-field Record and return Ok(Some(..)). fn main() { let schema = Arc::new(Schema { columns: vec!["line".into()] }); let mut reader = LinesReader { lines: vec!["third".into(), "second".into(), "first".into()], schema: schema.clone(), }; // The engine drives any FormatReader through exactly this loop: println!("schema: {:?}", reader.schema().unwrap()); loop { match reader.next_record() { Ok(Some(rec)) => println!("row: {:?}", rec.values), Ok(None) => { println!("end of input (clean)"); break; } Err(e) => { println!("error: {e:?}"); break; } } } }

💡 Hint 1

schema() is one line: clone the Arc and wrap it in Ok. For next_record, match self.lines.pop(): None is the clean end, Some(line) becomes a Record { schema: self.schema.clone(), values: vec![line] }.

Show solution

impl FormatReader for LinesReader {
    fn schema(&mut self) -> Result<Arc<Schema>, FormatError> {
        Ok(self.schema.clone())               // Arc::clone bumps a count, not a copy
    }
    fn next_record(&mut self) -> Result<Option<Record>, FormatError> {
        match self.lines.pop() {
            None => Ok(None),                 // clean end-of-stream
            Some(line) => Ok(Some(Record {
                schema: self.schema.clone(),
                values: vec![line],
            })),
        }
    }
}

That is a complete format implementation in miniature: the same two required methods, the same Ok(None)-means-done contract, the same shared Arc<Schema> as the real CsvReader. To put it in the engine you’d then register it (add the variant and the dispatch arm), which is the change-set the next section lists.

The change-set, and the round-trip test

To add a format end-to-end, there are three edit sites, and the predict-first sorted them for you, compiler-mandatory versus merely-necessary:

1. Add a variant (and its options struct) to InputFormat and/or OutputFormat in crates/clinker-plan/src/config/format.rs, and add its lowercase name to the format_name() match (also wildcard-free, so the compiler reminds you).
2. Implement FormatReader (and/or FormatWriter) in a new module under crates/clinker-format/src/<fmt>/, exactly the trait impl you just wrote in the faded rung. The compiler does not force this on its own; it’s necessary, not mandatory.
3. Add the dispatch arm in build_format_reader (executor/ingest.rs) and/or the writer dispatch in executor/registry.rs. These are the compiler-mandatory edits, the non-exhaustive-match error you triggered in the completion rung.

Prove it with a round-trip: read a sample, write it back, read again, assert every field survived, exactly what the CSV test does:

clinker-format ·writer.rs ·test_csv_roundtrip_lossless test @19acdcb4

#[test]
fn test_csv_roundtrip_lossless() {
    let input = "name,age,active\nAlice,30,true\nBob,25,false\n";
    // read -> records, write -> string, read again -> assert schemas + fields match
}

Solid on the format seam? Skip ahead →

Retrieval checkpoint

✓ Checkpoint: add a reader/writer/format

💡 Hint 1

Read build_format_reader to the end; there’s no _ => arm. Then read InputFormat: the #[serde(tag = "type")] attribute is the stage-1 string→variant mapping you don’t have to write.

💡 Hint 2

To feel the compiler-mandatory edit: add a dummy Tsv(Option<()>) variant to InputFormat, then run cargo check -p clinker-exec. The error points straight at build_format_reader (and format_name). Revert with git restore.

What the change establishes

A format is two trait impls (FormatReader/FormatWriter, two required methods each). A YAML type: string becomes a concrete reader in two stages: serde maps the string to an InputFormat/OutputFormat variant (config/format.rs), then a wildcard-free match in build_format_reader/build_format_writer (in clinker-exec) maps the variant to a Box<dyn Format*>. Adding the enum variant is the change the compiler forces you to complete at every dispatch site; writing the impl is necessary but not compiler-flagged; the dyn return type is the open seam your impl plugs into. Readers and writers dispatch through separate functions and separate enums.

// quick check

You add an InputFormat::Tsv variant but forget to add its arm to build_format_reader. What does the compiler do?

• Nothing; build_format_reader falls back to a default reader
• The build fails: build_format_reader's match is wildcard-free, so the new variant makes it non-exhaustive and the compiler points at the exact site to wire your reader
• It picks the Csv arm because Tsv is similar

// quick check

Why doesn't adding a new format require you to write any code that matches the format-name string?

• Because the CLI hard-codes every format name
• Because serde's #[serde(tag = "type", rename_all = "snake_case")] derives the string→variant mapping; you only add the variant and the dispatch arm
• Because format names are looked up in a runtime HashMap you must populate

// quick check

Spaced recall from the IO seam lesson: build_format_reader returns Box<dyn FormatReader> rather than a concrete CsvReader. Why a trait object here?

• To make the reader smaller in memory than the concrete type
• Because the format is chosen at run time from a YAML string, so the concrete type isn't known at compile time; dyn picks the right next_record through a vtable, and one return type covers every format
• Because traits can only ever be returned boxed

// quick check

An engineer says: 'I added the InputFormat::Tsv variant and the build_format_reader arm and it compiled, so the format is fully wired; the compiler would have caught anything missing.' Which is the MISCONCEPTION to correct?

• True: if it compiles with no _ => arm, every required site is wired
• The claim that a clean compile means fully wired: the FormatReader impl and the format_name() entry are needed too, and writing the impl in a new module is necessary but NOT something the compiler forces; only the exhaustive dispatch match is mandatory
• False: adding a variant never triggers any compiler error at all

You should be able to:

• You can name the two required methods on FormatReader and on FormatWriter, and say what Ok(None) from next_record means
• You implemented FormatReader for a from-scratch type with both required methods
• You can trace the two-stage path: serde maps the type: string to an InputFormat variant, then an exhaustive match maps the variant to a boxed reader
• You can list the three edit sites and distinguish the compiler-mandatory one from the merely-necessary ones

Verify in the checkout:

grep -n 'pub trait FormatReader\|pub trait FormatWriter' crates/clinker-format/src/traits.rs
grep -n 'pub enum InputFormat' crates/clinker-plan/src/config/format.rs
grep -n 'fn build_format_reader' crates/clinker-exec/src/executor/ingest.rs
cargo test -p clinker-format --offline test_csv_roundtrip_lossless -- --exact

Connections

You can now name the two format contracts, implement FormatReader for a new type, trace the string→variant→boxed-reader path, and sort the edit sites by which the compiler forces. Next: extend the engine’s middle by adding a new operator to the execution DAG, the change that touches the most places.

Prerequisites

• Add a CXL builtin

→

Leads to

• Add a transform/operator

Go deeper on the Rust (optional, one-directional, for traits and dyn taught from first principles, with no engine framing):

• Want traits and trait objects drilled from first principles in The Rust Book? Read the docs →

Glossary terms used: trait, trait object, dyn, seam.

Mark this lesson complete