Streaming records, not buffering — iterators

Clinker runs on files too big to hold in memory, so its readers never load the whole input. They hand the engine one record at a time and forget the last one. That streaming discipline is the engine’s entire bounded-memory promise: peak memory tracks the widest single record, not the size of the file. You met records back in Structs & provenance as “a Vec<Value> behind a schema”; this lesson opens both of those collections (the Vec of cells and the HashMap that turns a column name into a position) and then the Rust tool that makes one-at-a-time streaming work: the iterator. We lean on just enough of it to read the engine’s readers; The Rust Book, ch. 13 is the canonical treatment if you want iterators from first principles.

Prerequisites

• Provenance

→

Leads to

• Sharing & boxing

What you’ll be able to do

• Trace a record.get("status") lookup through the name→index→value path and name which collection does each step.
• Explain why the schema’s name→index map is stored once behind an Arc instead of once per record.
• Write a lazy iterator chain that filters and counts rows without building an intermediate Vec.
• Distinguish a lazy adapter (.filter, .map) from a consumer (.count, .collect) and predict when the chain actually runs.

New terms in this lesson (each is also expanded inline where you first need it):

• Vec<T>: a growable array indexed by position.
• iterator: produces a sequence one item at a time, on demand.
• lazy: nothing is computed until something pulls a value.
• adapter vs consumer: .filter/.map build a lazier chain; .count/.collect/for drive it.
• .collect(): drain an iterator into a Vec (materializes everything).

Predict first

✋ Predict first

A CSV reader streams a 1-billion-row file through an iterator chain: reader.filter(|r| r.is_active()).count(). Roughly how many records are live in memory at the peak of that run: one, a few, or all billion? Write down your answer and one sentence of reasoning before revealing.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

Roughly one record at a time: not a billion, and not zero.

.filter(...) is an adapter: it does not read the file. It hands back a new iterator that describes “yield only the active rows.” Nothing flows until .count(), a consumer, starts pulling. Each pull walks the chain once: the reader produces one record, filter tests it, count either adds 1 or drops it, and that record is freed before the next is pulled. The file’s size never enters the memory equation.

The wrong mental model is “filter builds the filtered list, then count measures it.” That would allocate a Vec of every active row, exactly the buffering this design exists to avoid. There is no intermediate list. Hold that: lazy means one at a time, and the rest of this lesson earns it.

A record’s two collections

clinker-record ·mod.rs ·Record type @19acdcb4

pub struct Record {
    schema: Arc<Schema>,   // column names + order, shared across rows
    values: Vec<Value>,    // the cells, indexed by position
    // ...
}

The cells are a Vec<Value>, a growable, contiguous array, indexed by position. But your CXL says status, a name. The bridge is in the Schema, which keeps a HashMap from column name to index:

clinker-record ·schema.rs ·Schema type @19acdcb4

pub struct Schema {
    columns: Vec<Box<str>>,           // the column names, in order
    field_metadata: Vec<Option<FieldMetadata>>,
    index: HashMap<Box<str>, usize>,  // name -> position
}

So record.get("status") is two steps: the schema’s HashMap turns "status" into, say, 4; then values[4] is the cell. Name → index → value. The HashMap does the first hop in O(1); the Vec does the second in O(1) by position. Neither step scans.

The schema is stored once and shared. That’s the Arc again, the exact sharing trick you met in Structs & provenance. A million rows from one file point at the same Schema, so the name→index map isn’t duplicated per row; cloning a record for the next row bumps the Arc’s count instead of copying the map.

A small but telling detail: the column names are stored as Box<str>, not String. A String carries a length and spare capacity for growth; these names never grow, so Box<str> drops the capacity field and stores only the bytes. The engine prefers the leaner type on the hot path, a habit you’ll see repeatedly.

Iterators: one record at a time

A reader doesn’t load the file and hand you a Vec<Record>. It is an iterator: it hands you records one at a time, on demand. Pulling the next item is next_record: give me the next row as Some(record), or None at the end.

That behavior is lazy: nothing is computed until asked, and only one record is held at a time. The laziness is a load-bearing property, not a style choice. A streaming pipeline keeps roughly one record’s worth of data live, whether the file is a thousand rows or a billion. Collecting the whole input into a Vec first would throw away the engine’s entire bounded-memory promise. (How the engine holds the line when an operation does need to accumulate, a sort or a join, is the subject of Execution & Memory.)

The pivotal distinction is adapter vs consumer. An adapter (.filter, .map, .take) returns another iterator and runs nothing. A consumer (.count, .collect, a for loop) is what finally pulls items through the chain. Read an iterator chain right to left for “what shape,” but remember the work happens only when the consumer on the end starts pulling.

Worked → completion → faded

You’ll build a lazy filter-and-count chain three times: fully worked, then with one gap, then from a skeleton. One new idea per rung.

Worked: a lazy chain that never buffers

Here is a self-contained chain over a Vec of pretend rows. Run it and read the comments: the adapters describe the work; count is the only thing that drives it.

rust // editable

reset ▶ run

fn main() {
  // Pretend these are records streaming from a source.
  let rows = vec!["active", "inactive", "active", "active"];

  // .iter() gives an iterator. .filter(...) is an ADAPTER: it returns
  // another iterator and runs NOTHING yet — it only describes the work.
  // .count() is the CONSUMER: it pulls each row through the chain once.
  let active = rows
      .iter()
      .filter(|status| **status == "active")
      .count();

  println!("active rows: {active}");

  // No intermediate Vec of active rows was ever built. Each row was
  // pulled, tested, counted, and dropped before the next was pulled.
}

fn main() { // Pretend these are records streaming from a source. let rows = vec!["active", "inactive", "active", "active"]; // .iter() gives an iterator. .filter(...) is an ADAPTER: it returns // another iterator and runs NOTHING yet — it only describes the work. // .count() is the CONSUMER: it pulls each row through the chain once. let active = rows .iter() .filter(|status| **status == "active") .count(); println!("active rows: {active}"); // No intermediate Vec of active rows was ever built. Each row was // pulled, tested, counted, and dropped before the next was pulled. }

Two * on status: .iter() yields &&str and the closure binds another &, so **status peels both references back to the &str to compare. The shape to take away is the chain itself: source iterator → lazy adapter → consumer.

Completion: fill the consumer

This chain is complete except for the consumer on the end. The goal: collect the active rows into an owned Vec<String> so a caller can keep them. Pick the method that drains an iterator into a collection, and tell it which collection.

rust // editable

reset ▶ run

fn main() {
  let rows = vec!["active", "inactive", "active", "active"];

  let active: Vec<String> = rows
      .iter()
      .filter(|status| **status == "active")
      .map(|status| status.to_string()) // own each &str as a String
      .______();                         // <- the consumer: drain into the Vec<String>

  println!("{active:?}"); // ["active", "active", "active"]
}

fn main() { let rows = vec!["active", "inactive", "active", "active"]; let active: Vec<String> = rows .iter() .filter(|status| **status == "active") .map(|status| status.to_string()) // own each &str as a String .______(); // <- the consumer: drain into the Vec<String> println!("{active:?}"); // ["active", "active", "active"] }

💡 Hint 1

The chain is still lazy until a consumer pulls it. You want the one named after gathering everything into a collection, and the target type is already declared as Vec<String>, so the consumer can infer where the items go.

Show solution

.collect()

.collect() is the consumer that materializes the whole iterator into a collection. The annotation let active: Vec<String> tells collect which collection to build. This is the deliberate opposite of the worked example: count kept memory flat, collect allocates a Vec holding every matching row. On the hot path the engine reaches for count-style consumers and for loops; collect is for when you genuinely need the whole list in hand.

Faded: write the whole chain

Your turn with a skeleton only. Write the body of count_active so it returns how many rows equal "active", using a lazy chain (filter then a consumer), building no intermediate Vec.

rust // editable

reset ▶ run

fn count_active(rows: &[&str]) -> usize {
  // your code here: iterate, filter to "active", and count — lazily.
  todo!()
}

fn main() {
  let rows = vec!["active", "inactive", "active", "active"];
  println!("{}", count_active(&rows)); // 3
}

fn count_active(rows: &[&str]) -> usize { // your code here: iterate, filter to "active", and count — lazily. todo!() } fn main() { let rows = vec!["active", "inactive", "active", "active"]; println!("{}", count_active(&rows)); // 3 }

Show solution

fn count_active(rows: &[&str]) -> usize {
    rows.iter()
        .filter(|status| **status == "active")
        .count()
}

.iter() borrows the slice and yields &&str; .filter is the lazy adapter; .count is the consumer that drives the single pass. No element is materialized into a new collection: the same bounded-memory shape the engine’s readers use on every record, scaled down to a slice you can run.

Why-bridge: laziness is the bounded-memory promise

Why does the engine go to the trouble of streaming records through lazy iterators instead of reading a file into a Vec<Record> and looping over it? Because the Vec version’s memory grows with the input, and the iterator version’s does not.

A reader that returns Vec<Record> must hold every row at once: a billion-row file means a billion live records, gigabytes resident, before a single one is processed. A reader that is an iterator holds one record, processes it, drops it, and pulls the next. Peak memory is set by the widest single record, not by the file. The adapters in between (.filter, .map) preserve that property because they’re lazy; they add no buffer of their own. Only a consumer that accumulates (.collect, a sort, a join) reintroduces growth, and Execution & Memory is where the engine spends real care bounding exactly those.

Apply: predict, then repair

This is a generation task, not recognition. The chain below is meant to be broken: it reuses an iterator after a consumer already drained it.

// WON'T COMPILE — kept here as a deliberate counterexample.
fn main() {
    let rows = vec!["active", "inactive", "active"];
    let chain = rows.iter().filter(|s| **s == "active");

    let n = chain.count();      // consumer #1 drains the iterator (moves it)
    let kept: Vec<_> = chain.collect(); // <- error here: chain was moved
    println!("{n} {kept:?}");
}

Predict the compiler first. What error does the second line produce, and why?

Show the compiler's answer

error[E0382]: use of moved value: `chain`
 --> src/main.rs
  |
  |     let n = chain.count();
  |                   ------- `chain` moved due to this method call
  |     let kept: Vec<_> = chain.collect();
  |                        ^^^^^ value used here after move

A consumer like .count() takes the iterator by value and drives it to exhaustion; there’s nothing left to pull, and the iterator is gone (moved). You can’t consume the same chain twice. The fix is to build the chain once per consumer:

let kept: Vec<_> = rows.iter().filter(|s| **s == "active").collect();
let n = kept.len(); // count from the materialized Vec, or rebuild the chain

This is the lazy/consumer split made visible by the borrow checker: adapters are cheap to re-describe, but a consumer ends the iterator’s life. One chain, one drive.

Retrieval checkpoint

✓ Checkpoint

// quick check

How does looking up the field named status find the right cell when cells are stored by position?

• It scans every cell comparing names
• The schema's HashMap maps the name to an index, then values[index] is the cell
• Each Value stores its own column name

// quick check

Why does the engine pull records one at a time instead of reading the whole file into a Vec<Record>?

• It's the only way Rust can read files
• To keep memory roughly constant (about one record live at a time) regardless of file size
• Because Vec has a size limit

// quick check

Two engineers describe rows.iter().filter(|r| r.active()).count(). Which statement is the MISCONCEPTION to correct?

• .filter is lazy: it returns another iterator and runs nothing until .count() pulls each row through
• .filter immediately builds a Vec of the active rows, and then .count() measures that Vec's length
• .count() is the consumer that drives the single pass and takes the iterator by value

// quick check

Spaced recall from the provenance lesson: a million rows from customers.csv each carry the same source filename in provenance. How is that filename stored so it isn't copied a million times?

• As a plain String cloned into each record
• Behind an Arc<str> shared handle: every record points at the same one allocation, and cloning only bumps the count
• It's recomputed from the row index each time

You should be able to:

• You can trace record.get("status") through name → index → value and name the HashMap and Vec steps
• You can explain why the schema's name→index map lives once behind an Arc, not once per record
• You wrote a lazy filter-then-count chain that builds no intermediate Vec
• You can tell a lazy adapter (.filter/.map) from a consumer (.count/.collect) and say when the chain runs
• You predicted the 'use of moved value' error from consuming one iterator twice

Verify in the checkout:

grep -nA4 'pub struct Record' crates/clinker-record/src/record/mod.rs
grep -nA4 'pub struct Schema' crates/clinker-record/src/schema.rs

Connections

You know the two collections inside a record, the Vec<Value> of cells and the schema’s name→index HashMap, and why readers stream them lazily through iterators rather than collecting them. Next: the smart pointers Box, Rc, and Arc that the engine uses to share data like that schema without copying it.

Prerequisites

• Provenance

→

Leads to

• Sharing & boxing

Go deeper on the Rust (optional, the same Vec and its move/clone cost taught from first principles in The Rust Book):

• Want Vec's ownership and the cost of cloning it taught from first principles? Read the docs →

Glossary terms used: Vec, iterator, lazy, adapter, collect, Arc.

Mark this lesson complete