Add a reader/writer/format

In lesson 3.1 you met the IO seam from the outside: formats are an open, runtime-chosen plug-in surface, so the engine reaches them through Box<dyn FormatReader>. Now you’re on the inside, adding a format. This lesson shows the full change-set — and it pulls together two earlier ideas: the dyn seam (3.1) is where your code plugs in, and the closed exhaustive match (4.1) is what forces every dispatch site to acknowledge your new format.

Prerequisites

• Add a CXL builtin

→

Leads to

• Add a transform/operator

You’ll be able to: implement the FormatReader/FormatWriter traits, explain the two-stage path from a YAML type: string to a concrete reader, and list the sites a new format touches — and which one the compiler makes mandatory.

Two traits, two required methods each

A format is whatever can stream Records in and out. The contracts are tiny:

clinker-format ·traits.rs ·FormatReader trait @47d2e12

/// Streaming record reader. Yields records one at a time.
///
/// `&mut self` on `schema()` because some formats (e.g. CSV) must read
/// the first row to discover column names. Must be `Send` for executor
/// ownership transfer; not `Sync` — single-threaded streaming.
pub trait FormatReader: Send {
    fn schema(&mut self) -> Result<Arc<Schema>, FormatError>;
    fn next_record(&mut self) -> Result<Option<Record>, FormatError>;
    // ...plus a few defaulted methods for envelopes / multi-file readers
}

clinker-format ·traits.rs ·FormatWriter trait @47d2e12

/// Streaming record writer. Consumes records one at a time.
pub trait FormatWriter: Send {
    fn write_record(&mut self, record: &Record) -> Result<(), FormatError>;
    fn flush(&mut self) -> Result<(), FormatError>;
    // ...plus defaulted begin_document / end_document for envelope framing
}

Two required methods on each, both fallible, both streaming. next_record returning Ok(None) means end-of-stream. The : Send bound is the same one lesson 4.2 explained: the executor moves readers onto worker threads, so a reader must be Send. A minimal new format implements just those two-each methods; the defaulted ones (envelopes, multi-file) are opt-in.

A concrete reader: CSV

Here’s the shape of a real implementation — CsvReader, the simplest complete one:

clinker-format ·reader.rs ·CsvReader type @47d2e12

pub struct CsvReader<R: Read> {
    inner: csv::Reader<SkipBom<R>>,
    schema: Option<Arc<Schema>>,    // discovered from the header, cached
    config: CsvReaderConfig,
    row_count: u64,
    record_buf: csv::StringRecord,
}

impl<R: Read + Send> FormatReader for CsvReader<R> {
    fn schema(&mut self) -> Result<Arc<Schema>, FormatError> {
        self.ensure_schema()                  // reads the header row once
    }
    fn next_record(&mut self) -> Result<Option<Record>, FormatError> {
        let schema = self.ensure_schema()?;
        if !self.inner.read_record(&mut self.record_buf)? {
            return Ok(None);                  // end of stream
        }
        let values = self.record_buf.iter().map(|f| Value::String(f.into())).collect();
        Ok(Some(Record::new(schema, values)))
    }
}

That’s the whole pattern: hold the underlying byte source plus a cached Arc<Schema>, and have next_record pull one row and emit Ok(Some(Record)) until the source is dry. The impl is generic over R: Read + Send, not tied to files.

From a YAML string to your reader, in two stages

A pipeline says type: csv in YAML. How does that string reach CsvReader? Not by a string-match you write — it’s a two-stage hop, and understanding it tells you exactly what to edit.

Stage 1 — string to enum variant, by serde. The config layer has a closed enum of known formats, adjacently tagged so serde maps type: csv to a variant automatically:

clinker-plan ·format.rs ·InputFormat type @47d2e12

#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type", content = "options", rename_all = "snake_case")]
pub enum InputFormat {
    Csv(Option<CsvInputOptions>),
    Json(Option<JsonInputOptions>),
    Xml(Option<XmlInputOptions>),
    FixedWidth(Option<FixedWidthInputOptions>),
    Edifact(Option<EdifactInputOptions>),
    X12(Option<X12InputOptions>),
    Hl7(Option<Hl7InputOptions>),
    Swift(Option<SwiftInputOptions>),
}

The #[serde(tag = "type", rename_all = "snake_case")] is what turns the string "csv" into the variant InputFormat::Csv. You never write name-matching code.

Stage 2 — enum variant to boxed reader, by an exhaustive match. In the executor, one function matches the variant and returns the boxed trait object:

clinker-exec ·ingest.rs ·build_format_reader fn @47d2e12

fn build_format_reader(
    input: &clinker_plan::config::SourceConfig,
    source: ReopenableSource,
) -> Result<Box<dyn FormatReader>, PipelineError> {
    match &input.format {
        InputFormat::Csv(opts)  => Ok(Box::new(CsvReader::from_reader(/* ... */))),
        InputFormat::Json(opts) => Ok(Box::new(JsonReader::from_source(/* ... */)?)),
        InputFormat::Xml(opts)  => { /* ... */ }
        // ...one arm per variant, NO `_ =>` catch-all
    }
}

This is the lesson-4.1 payoff in action: the match is wildcard-free, so the moment you add a variant to InputFormat, this function stops compiling until you add its arm. The open dyn seam (return type Box<dyn FormatReader>) and the closed enum dispatch (match with no wildcard) cooperate — dyn lets your reader plug in; the exhaustive enum makes the compiler hand you the list of sites to wire.

Two registries, not one

Writers are dispatched the same way but by a separate function (build_format_writer, matching a separate OutputFormat enum). There’s also a narrower FormatKind enum (four variants) used elsewhere — it is not the dispatch key; don’t wire a new format through it. Reader dispatch and writer dispatch are two independent exhaustive matches.

The exhaustive seam, in miniature

Here’s the enum-match-to-trait-object pattern on its own. Add a Tsv variant to Format and the build_reader match below stops compiling — the compiler names the exact site:

rust // editable

reset ▶ run

trait Reader {
  fn next_line(&mut self) -> Option<String>;
}

struct CsvReader { rows: Vec<String> }
impl Reader for CsvReader {
  fn next_line(&mut self) -> Option<String> { self.rows.pop() }
}

struct JsonReader { rows: Vec<String> }
impl Reader for JsonReader {
  fn next_line(&mut self) -> Option<String> { self.rows.pop() }
}

enum Format { Csv, Json }   // add Tsv here ...

// ... and this wildcard-free match stops compiling until you add the arm.
fn build_reader(fmt: Format) -> Box<dyn Reader> {
  match fmt {
      Format::Csv  => Box::new(CsvReader { rows: vec!["a,b".into()] }),
      Format::Json => Box::new(JsonReader { rows: vec!["{}".into()] }),
  }
}

fn main() {
  for fmt in [Format::Csv, Format::Json] {   // construct both variants
      let mut r = build_reader(fmt);
      while let Some(line) = r.next_line() {
          println!("{line}");
      }
  }
}

trait Reader { fn next_line(&mut self) -> Option<String>; } struct CsvReader { rows: Vec<String> } impl Reader for CsvReader { fn next_line(&mut self) -> Option<String> { self.rows.pop() } } struct JsonReader { rows: Vec<String> } impl Reader for JsonReader { fn next_line(&mut self) -> Option<String> { self.rows.pop() } } enum Format { Csv, Json } // add Tsv here ... // ... and this wildcard-free match stops compiling until you add the arm. fn build_reader(fmt: Format) -> Box<dyn Reader> { match fmt { Format::Csv => Box::new(CsvReader { rows: vec!["a,b".into()] }), Format::Json => Box::new(JsonReader { rows: vec!["{}".into()] }), } } fn main() { for fmt in [Format::Csv, Format::Json] { // construct both variants let mut r = build_reader(fmt); while let Some(line) = r.next_line() { println!("{line}"); } } }

The return type Box<dyn Reader> is the open seam; the wildcard-free match is the closed dispatch. That’s the same division of labor as the real build_format_reader.

The change-set, and the round-trip test

To add a format end-to-end:

1. Add a variant (and its options struct) to InputFormat and/or OutputFormat in crates/clinker-plan/src/config/format.rs, and add its lowercase name to the format_name() match (also wildcard-free — the compiler reminds you).
2. Implement FormatReader (and/or FormatWriter) in a new module under crates/clinker-format/src/<fmt>/.
3. Add the dispatch arm in build_format_reader (executor/ingest.rs) and/or the writer dispatch in executor/registry.rs. These are the compiler-mandatory edits.

Prove it with a round-trip: read a sample, write it back, read again, assert every field survived — exactly what the CSV test does:

clinker-format ·writer.rs ·test_csv_roundtrip_lossless test @47d2e12

#[test]
fn test_csv_roundtrip_lossless() {
    let input = "name,age,active\nAlice,30,true\nBob,25,false\n";
    // read -> records, write -> string, read again -> assert schemas + fields match
}

// quick check

You add an InputFormat::Tsv variant but forget to add its arm to build_format_reader. What does the compiler do?

• Nothing — build_format_reader falls back to a default reader
• The build fails: build_format_reader's match is wildcard-free, so the new variant makes it non-exhaustive and the compiler points at the exact site to wire your reader
• It picks the Csv arm because Tsv is similar

Solid on the format seam? Skip ahead →

Wire one yourself

✓ Checkpoint — add a reader/writer/format

💡 Hint 1

Read build_format_reader to the end — there’s no _ => arm. Then read InputFormat: the #[serde(tag = "type")] attribute is the stage-1 string→variant mapping you don’t have to write.

💡 Hint 2

To feel the compiler-mandatory edit: add a dummy Tsv(Option<()>) variant to InputFormat, then run cargo check -p clinker-exec. The error points straight at build_format_reader (and format_name). Revert with git restore.

What the change establishes

A format is two trait impls (FormatReader/FormatWriter, two required methods each). A YAML type: string becomes a concrete reader in two stages: serde maps the string to an InputFormat/OutputFormat variant (config/format.rs), then a wildcard-free match in build_format_reader/build_format_writer (in clinker-exec) maps the variant to a Box<dyn Format*>. Adding the enum variant is the change the compiler forces you to complete at every dispatch site; the dyn return type is the open seam your impl plugs into. Readers and writers dispatch through separate functions and separate enums.

// quick check

Why doesn't adding a new format require you to write any code that matches the format-name string?

• Because the CLI hard-codes every format name
• Because serde's #[serde(tag = "type", rename_all = "snake_case")] derives the string→variant mapping; you only add the variant and the dispatch arm
• Because format names are looked up in a runtime HashMap you must populate

You should be able to:

• You can name the two required methods on FormatReader and on FormatWriter
• You can explain the two-stage path: serde maps the type: string to an InputFormat variant, then an exhaustive match maps the variant to a boxed reader
• You can list the three edit sites and say which one the compiler forces

Verify in the checkout:

grep -n 'pub trait FormatReader\|pub trait FormatWriter' crates/clinker-format/src/traits.rs
grep -n 'pub enum InputFormat' crates/clinker-plan/src/config/format.rs
grep -n 'fn build_format_reader' crates/clinker-exec/src/executor/ingest.rs
cargo test -p clinker-format --offline test_csv_roundtrip_lossless -- --exact

You’ve extended the engine’s edges. Next: extend its middle — add a new operator to the execution DAG, the change that touches the most places.

Mark this lesson complete