The IO seam — traits

Clinker reads and writes eight wildly different file formats (CSV, JSON, XML, fixed-width, EDIFACT, X12, HL7, SWIFT), yet the executor that pushes records through the DAG contains zero format-specific code. The mechanism that makes that possible is the engine’s most important seam: a single contract every format satisfies, so a new format plugs in without the execution loop changing by a character. You spent Data & Representation taking the data layer apart cell by cell; now we move up a layer to where new behaviour plugs in. The Rust tool underneath the seam is the trait; we use just enough of it to read the real FormatReader, and The Rust Book, ch. 10 is the canonical treatment if you want traits from first principles.

Prerequisites

• Zero-copy reads

→

Leads to

• One reader, every format

What you’ll be able to do

• Read the FormatReader trait and name its two required trait methods and what a full drain looks like.
• Explain how every format reaches the rest of the engine through that one contract, with zero format-specific code in the executor.
• Write a trait, two implementors, and a function that drives any implementor through the trait: runnable Rust, the engine’s seam in miniature.
• Distinguish a trait object (Box<dyn FormatReader>) from a concrete type, and say why the engine boxes readers and what dynamic dispatch costs here.

New terms in this lesson (each is also expanded inline at the point you first need it):

• trait: a contract of methods, not a type.
• trait method: a method the contract requires (or supplies by default).
• impl: the block where a concrete type satisfies the trait.
• seam: the plug-in boundary a trait creates.
• trait object (Box<dyn FormatReader>): pre-taught where it appears below.

Predict first

✋ Predict first

A function takes one parameter: drain(reader: &mut dyn FormatReader). You call it once with a CSV reader and once with a JSON reader: two different structs, with two different next_record bodies. Does drain need to change between the two calls, or even know which one it got? Write down your answer and one sentence of why before revealing.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

No. drain neither changes nor learns which reader it got. That is the whole point of the seam, and it is the skill this lesson builds.

drain is typed against the trait FormatReader, not against CsvReader or JsonReader. It can call next_record() because the contract guarantees that method exists on anything you pass; which concrete next_record runs is resolved for it at run time. Add a fourth format tomorrow and drain doesn’t change by a character.

The trap is imagining drain must contain a match on the format: “if CSV do this, if JSON do that.” That is exactly the format-specific code the trait exists to abolish. The executor that drives all eight of clinker’s formats contains zero such branches. Hold that prediction; everything below earns it.

The motivating problem: one engine, eight formats

Clinker ships readers and writers for CSV, JSON, XML, fixed-width, EDIFACT, X12, HL7, and SWIFT. That’s eight wildly different ways to lay bytes on disk. Yet the executor, the part that pushes records through the DAG, contains zero format-specific code. It never asks “is this a CSV?” How can the same execution loop drive all eight?

The answer is a trait: a contract that says what a reader can do without fixing which reader it is. Every format implements the same two methods, and the engine talks only to the contract.

clinker-format ·traits.rs ·FormatReader trait @19acdcb4

/// Streaming record reader. Yields records one at a time.
pub trait FormatReader: Send {
    fn schema(&mut self) -> Result<Arc<Schema>, FormatError>;
    fn next_record(&mut self) -> Result<Option<Record>, FormatError>;
    // ... plus default-bodied methods for multi-file / envelope handling
}

Two required trait methods carry the whole seam: ask for the schema, then pull records until next_record returns Ok(None). A FormatReader is exactly “a thing the engine can ask for a schema and then drain, one Record at a time.” Notice the return type of next_record, Result<Option<Record>, FormatError>: it’s the composed three-outcome type you read in Data & Representation, and the trait just promises every format produces it. The write side is its mirror image, consuming records one at a time and flushing:

clinker-format ·traits.rs ·FormatWriter trait @19acdcb4

/// Streaming record writer. Consumes records one at a time.
pub trait FormatWriter: Send {
    fn write_record(&mut self, record: &Record) -> Result<(), FormatError>;
    fn flush(&mut self) -> Result<(), FormatError>;
    // ... plus default-bodied document-framing methods
}

Notice the supertrait bound : Send. The doc comment is explicit about why: a reader is moved onto the executor’s per-source ingest thread, so it must be Send. It is deliberately not Sync: a single reader is driven by one thread, streaming, never shared. That bound is a small architectural decision encoded in the type.

A minimal model: the trait is the seam

Strip the engine away and the pattern is small. A trait with one method, two implementors, and a function that works on any implementor. This is the same shape as FormatReader, shrunk to something you can run:

rust // editable

reset ▶ run

trait FormatReader {
  fn next_record(&mut self) -> Option<String>;
}

struct CsvReader { rows: Vec<String> }
struct JsonReader { left: u32 }

impl FormatReader for CsvReader {
  fn next_record(&mut self) -> Option<String> {
      self.rows.pop()
  }
}
impl FormatReader for JsonReader {
  fn next_record(&mut self) -> Option<String> {
      if self.left == 0 { return None; }
      self.left -= 1;
      Some(format!("json row {}", self.left))
  }
}

// Drains ANY reader — it knows the trait, not the type.
fn drain(reader: &mut dyn FormatReader) -> usize {
  let mut n = 0;
  while reader.next_record().is_some() { n += 1; }
  n
}

fn main() {
  let mut csv = CsvReader { rows: vec!["a".into(), "b".into()] };
  let mut json = JsonReader { left: 3 };
  println!("csv rows:  {}", drain(&mut csv));
  println!("json rows: {}", drain(&mut json));
}

trait FormatReader { fn next_record(&mut self) -> Option<String>; } struct CsvReader { rows: Vec<String> } struct JsonReader { left: u32 } impl FormatReader for CsvReader { fn next_record(&mut self) -> Option<String> { self.rows.pop() } } impl FormatReader for JsonReader { fn next_record(&mut self) -> Option<String> { if self.left == 0 { return None; } self.left -= 1; Some(format!("json row {}", self.left)) } } // Drains ANY reader — it knows the trait, not the type. fn drain(reader: &mut dyn FormatReader) -> usize { let mut n = 0; while reader.next_record().is_some() { n += 1; } n } fn main() { let mut csv = CsvReader { rows: vec!["a".into(), "b".into()] }; let mut json = JsonReader { left: 3 }; println!("csv rows: {}", drain(&mut csv)); println!("json rows: {}", drain(&mut json)); }

drain is the executor in miniature: it takes &mut dyn FormatReader and never learns whether it’s draining CSV or JSON, exactly what you predicted. That, and nothing more, is the seam doing its job: one function written against the contract, indifferent to which implementor it drives.

Worked → completion → faded

You just read the seam. Now you build one, scaffolded down to writing the whole thing yourself. Each rung adds one idea: first see a full trait + two impls + a driver; then supply one missing impl body; then write a fresh trait and its driver from a skeleton.

Worked: a trait, two impls, and a driver that drives both

Here is a complete, runnable seam. A trait Counter with one required method, two unrelated structs that each impl it, and a total function that drives any Counter through the contract. Run it and watch one function tally two different structs.

rust // editable

reset ▶ run

// The contract: anything that is a Counter can yield its next count, or None when done.
trait Counter {
  fn next_count(&mut self) -> Option<u32>;
}

// Implementor 1: walks a fixed list of numbers.
struct ListCounter { items: Vec<u32> }
impl Counter for ListCounter {
  fn next_count(&mut self) -> Option<u32> {
      self.items.pop()
  }
}

// Implementor 2: counts DOWN from a starting value.
struct Countdown { remaining: u32 }
impl Counter for Countdown {
  fn next_count(&mut self) -> Option<u32> {
      if self.remaining == 0 { return None; }
      self.remaining -= 1;
      Some(self.remaining)
  }
}

// The driver: speaks the trait, not the type. Sums until the source is drained.
fn total(c: &mut dyn Counter) -> u32 {
  let mut sum = 0;
  while let Some(n) = c.next_count() {
      sum += n;
  }
  sum
}

fn main() {
  let mut list = ListCounter { items: vec![10, 20, 30] };
  let mut down = Countdown { remaining: 4 };
  println!("list total: {}", total(&mut list)); // 60
  println!("down total: {}", total(&mut down)); // 3 + 2 + 1 + 0 = 6
}

// The contract: anything that is a Counter can yield its next count, or None when done. trait Counter { fn next_count(&mut self) -> Option<u32>; } // Implementor 1: walks a fixed list of numbers. struct ListCounter { items: Vec<u32> } impl Counter for ListCounter { fn next_count(&mut self) -> Option<u32> { self.items.pop() } } // Implementor 2: counts DOWN from a starting value. struct Countdown { remaining: u32 } impl Counter for Countdown { fn next_count(&mut self) -> Option<u32> { if self.remaining == 0 { return None; } self.remaining -= 1; Some(self.remaining) } } // The driver: speaks the trait, not the type. Sums until the source is drained. fn total(c: &mut dyn Counter) -> u32 { let mut sum = 0; while let Some(n) = c.next_count() { sum += n; } sum } fn main() { let mut list = ListCounter { items: vec![10, 20, 30] }; let mut down = Countdown { remaining: 4 }; println!("list total: {}", total(&mut list)); // 60 println!("down total: {}", total(&mut down)); // 3 + 2 + 1 + 0 = 6 }

total is drain again: one driver, written once against Counter, driving two structs it never names.

Completion: supply the missing impl body

Below, the trait, the driver, and one implementor are done. The second implementor, EvenStream, which should yield 0, 2, 4, … up to a limit, is missing its next_count body. Fill in only that one method so the program drains cleanly to None.

rust // editable

reset ▶ run

trait Counter {
  fn next_count(&mut self) -> Option<u32>;
}

struct Countdown { remaining: u32 }
impl Counter for Countdown {
  fn next_count(&mut self) -> Option<u32> {
      if self.remaining == 0 { return None; }
      self.remaining -= 1;
      Some(self.remaining)
  }
}

// Yields 0, 2, 4, ... and stops once it would reach or pass `limit`.
struct EvenStream { current: u32, limit: u32 }
impl Counter for EvenStream {
  fn next_count(&mut self) -> Option<u32> {
      // TODO: return None once self.current >= self.limit; otherwise
      // hand back the current even number and advance by 2.
      todo!()
  }
}

fn total(c: &mut dyn Counter) -> u32 {
  let mut sum = 0;
  while let Some(n) = c.next_count() { sum += n; }
  sum
}

fn main() {
  let mut evens = EvenStream { current: 0, limit: 8 };
  println!("evens total: {}", total(&mut evens)); // 0 + 2 + 4 + 6 = 12
}

trait Counter { fn next_count(&mut self) -> Option<u32>; } struct Countdown { remaining: u32 } impl Counter for Countdown { fn next_count(&mut self) -> Option<u32> { if self.remaining == 0 { return None; } self.remaining -= 1; Some(self.remaining) } } // Yields 0, 2, 4, ... and stops once it would reach or pass `limit`. struct EvenStream { current: u32, limit: u32 } impl Counter for EvenStream { fn next_count(&mut self) -> Option<u32> { // TODO: return None once self.current >= self.limit; otherwise // hand back the current even number and advance by 2. todo!() } } fn total(c: &mut dyn Counter) -> u32 { let mut sum = 0; while let Some(n) = c.next_count() { sum += n; } sum } fn main() { let mut evens = EvenStream { current: 0, limit: 8 }; println!("evens total: {}", total(&mut evens)); // 0 + 2 + 4 + 6 = 12 }

💡 Hint 1

An implementor’s job is to honour the trait’s signature: return None when the source is exhausted, otherwise Some(value). Here “exhausted” means current has reached limit. Don’t forget to advance current so the next call makes progress.

Show solution

fn next_count(&mut self) -> Option<u32> {
    if self.current >= self.limit {
        return None;
    }
    let n = self.current;
    self.current += 2;
    Some(n)
}

total never changed. You taught a brand-new type to satisfy the same contract, and the driver drove it untouched; the seam absorbed the new implementor exactly the way the engine absorbs a new format.

Faded: write the whole seam yourself

Now with no method bodies given. Define a trait Source with one required method next_line(&mut self) -> Option<String>; give it two implementors, Fixed (yields lines from a Vec<String>) and Repeat (yields the same string times times), and a driver collect_all(&mut dyn Source) -> Vec<String> that drains any source into a Vec. The main is written for you and asserts the expected output.

rust // editable

reset ▶ run

// TODO: define `trait Source` with one required method:
//   fn next_line(&mut self) -> Option<String>;

// TODO: struct Fixed { lines: Vec<String> }  — pop from `lines`.

// TODO: struct Repeat { text: String, times: u32 } — yield `text` `times` times, then None.

// TODO: fn collect_all(src: &mut dyn Source) -> Vec<String>
//   drain the source, pushing each line into a Vec, return it.

fn main() {
  let mut fixed = Fixed { lines: vec!["b".into(), "a".into()] };
  let mut repeat = Repeat { text: "x".into(), times: 3 };
  // Fixed pops from the end, so this comes out ["a", "b"].
  assert_eq!(collect_all(&mut fixed), vec!["a", "b"]);
  assert_eq!(collect_all(&mut repeat), vec!["x", "x", "x"]);
  println!("both sources drained through one driver");
}

// TODO: define `trait Source` with one required method: // fn next_line(&mut self) -> Option<String>; // TODO: struct Fixed { lines: Vec<String> } — pop from `lines`. // TODO: struct Repeat { text: String, times: u32 } — yield `text` `times` times, then None. // TODO: fn collect_all(src: &mut dyn Source) -> Vec<String> // drain the source, pushing each line into a Vec, return it. fn main() { let mut fixed = Fixed { lines: vec!["b".into(), "a".into()] }; let mut repeat = Repeat { text: "x".into(), times: 3 }; // Fixed pops from the end, so this comes out ["a", "b"]. assert_eq!(collect_all(&mut fixed), vec!["a", "b"]); assert_eq!(collect_all(&mut repeat), vec!["x", "x", "x"]); println!("both sources drained through one driver"); }

💡 Hint 1

The driver doesn’t care which implementor it has: write it once against &mut dyn Source, looping while let Some(line) = src.next_line(). Each impl Source for … block supplies a next_line that returns None when its own source is exhausted.

Show solution

trait Source {
    fn next_line(&mut self) -> Option<String>;
}

struct Fixed { lines: Vec<String> }
impl Source for Fixed {
    fn next_line(&mut self) -> Option<String> {
        self.lines.pop()
    }
}

struct Repeat { text: String, times: u32 }
impl Source for Repeat {
    fn next_line(&mut self) -> Option<String> {
        if self.times == 0 { return None; }
        self.times -= 1;
        Some(self.text.clone())
    }
}

fn collect_all(src: &mut dyn Source) -> Vec<String> {
    let mut out = Vec::new();
    while let Some(line) = src.next_line() {
        out.push(line);
    }
    out
}

You wrote the seam end to end: one contract, two unrelated implementors, one driver that drives either. Scale next_line up to next_record, swap your two structs for eight format readers, and you have FormatReader and the executor’s drain loop. The shape does not change.

Why a trait object, and what dyn costs

The format a job uses isn’t known when the engine is compiled; it’s chosen at run time, from the plan. So the executor needs a single variable that can hold any reader. That’s a trait object: Box<dyn FormatReader>, a heap-allocated value plus a hidden pointer to a vtable (the table of “which next_record does this actual reader use?”). Calling through it is dynamic dispatch: the concrete method is looked up at run time.

In real clinker, the concrete readers are themselves generic structs: CsvReader<R>, FixedWidthReader<R>, and so on, generic over the byte source R. They’re built type-specifically, then immediately boxed into a trait object at one factory function so that everything downstream is uniform:

clinker-exec ·mod.rs ·RecordSource trait @19acdcb4

// crates/clinker-exec/src/executor/ingest.rs — the dispatch boundary
fn build_format_reader(/* … */) -> Box<dyn FormatReader> {
    match &input.format {
        InputFormat::Csv(opts)  => Box::new(CsvReader::new(/* … */)),
        InputFormat::Json(opts) => Box::new(JsonReader::new(/* … */)),
        // ... one arm per format, each boxed into the same trait-object type
    }
}

This is worth pausing on, because it’s the shape of every plug-in seam in clinker: a typed enum (InputFormat) is matched once at the boundary, each arm constructs the right concrete reader, and all of them collapse into one Box<dyn FormatReader>. There is no string-keyed “format registry”: dispatch is over the closed enum you met in Data & Representation, so an unknown format is a plan-time error, not a runtime lookup miss.

What does dyn cost? One pointer-indirection per call. Here that’s invisible: a vtable call is dwarfed by the file IO and parsing that produce each record. Dynamic dispatch is the right trade exactly where the set of types is open-ended and chosen at run time, and the per-call cost is noise against the work being dispatched. The next lesson, One reader, every format, shows the opposite choice, generics, for the field-read hot path where that same pointer-indirection would not be noise.

One step further: the transport generalization

FormatReader assumes bytes; it decodes a stream. But a SQL SELECT cursor yields rows with no byte body at all. So one crate up, the executor defines a broader contract, RecordSource, and bridges the file case to it with a single blanket impl:

crates/clinker-exec/src/source/mod.rs

impl RecordSource for Box<dyn FormatReader> {
    // a file transport reaches the row-oriented seam by wrapping its byte reader
}

You don’t need the details yet; Execution & Memory returns to it. The point is the layering: a narrow, byte-oriented trait (FormatReader) nested inside a broader, transport-agnostic one (RecordSource), each a clean seam. Traits compose into layers the same way types do.

Comfortable with traits and trait objects already? Skip ahead →

Retrieval checkpoint

✓ Checkpoint: the IO seam

💡 Hint 1

Start at the trait definition in traits.rs, then follow build_format_reader in crates/clinker-exec/src/executor/ingest.rs to see where concrete readers are boxed.

What the grep tour shows

The first two greps land on the trait declarations: two required methods each, both with a : Send supertrait. The third shows the factory: a match over the InputFormat enum whose arms each Box::new(...) a concrete reader into the shared Box<dyn FormatReader> return type. That single function is the entire dynamic-dispatch boundary for ingest.

// quick check

A new format, Avro, is added. Which existing code must change to teach the executor to read it?

• Every function that drains records
• Only the format crate (a new reader implementing FormatReader) plus the one match arm in the reader factory; the executor's drain loop is untouched
• The Record and Value types

// quick check

Why does the engine store readers as Box<dyn FormatReader> instead of a concrete reader type?

• The format is chosen at run time, so the executor needs one variable that can hold any reader; dynamic dispatch lets every format share a single type
• Trait objects always run faster than concrete types
• Because a trait can only be used through a Box

// quick check

MISCONCEPTION CHECK: an engineer says 'we box readers into Box<dyn FormatReader> for speed, since dynamic dispatch is faster than calling a concrete method.' What is wrong with that?

• Nothing; vtable calls are the fastest form of dispatch
• Dynamic dispatch is not chosen for speed: it adds a pointer-indirection per call. It's chosen because the set of formats is open-ended and picked at run time, and that per-call cost is noise against file IO
• Boxing is required by the compiler whenever a trait has more than one method

// quick check

Spaced recall from the Option and Result lesson: FormatReader::next_record returns Result<Option<Record>, FormatError>. What does Ok(None) mean?

• A read error occurred
• A clean end of input: the finite source is exhausted, so the drain loop stops with no error
• An empty record carrying no fields

You should be able to:

• You can name the two required methods of FormatReader and what a full drain looks like
• You can explain how a new format reaches the engine without the executor's drain loop changing
• You wrote a trait, two implementors, and a driver that drives either through the contract
• You can explain why readers are stored as trait objects and what dynamic dispatch costs here
• You can explain why the : Send bound is on the trait but : Sync is not

Verify in the checkout:

grep -n 'pub trait FormatReader' crates/clinker-format/src/traits.rs
grep -n 'pub trait FormatWriter' crates/clinker-format/src/traits.rs
grep -rn 'Box<dyn FormatReader>' crates/clinker-exec/src/executor/ingest.rs

Connections

You’ve seen the engine’s open seam: a trait, implemented per format, boxed into a trait object at one boundary so the rest of the engine stays format-blind. You also built one from scratch (a contract, two implementors, one driver) which is the same shape, scaled down. Next, One reader, every format: the other dispatch strategy, generics, and why the field-read hot path makes the opposite choice.

Prerequisites

• Zero-copy reads

→

Leads to

• One reader, every format

Go deeper on the Rust (optional, one-directional, for traits taught from first principles in The Rust Book):

• Want traits from first principles, on a two-backing toy with no engine framing? Read the docs →

Glossary terms used: trait, trait method, impl, seam, trait object.

Mark this lesson complete