What is a record?

Back in Orientation you ran customer_etl and Clinker reported “5 records.” A Record’s Journey follows one of those records all the way through the engine. But first: what is a record? It’s the unit of data that flows through every pipeline, and it’s built from three vocabulary types you’ll see everywhere: a cell, a row, and the columns that name the cells.

What you’ll be able to do

Name the three types that make up a record (Value is a cell, Record a row, Schema the columns) and say what each one holds.
Explain why a freshly-read CSV record is entirely Value::String, and what later turns a string into a number.
Trace how a cell is located: how values[i] finds the right column through the shared Schema.
Write a miniature record (a schema plus a positional Vec of values) and print each cell against its column name.

(We meet these types shallowly here; Data & Representation opens each one up: the optimized string type, the Arc sharing, the per-record context.)

New terms in this lesson (each is named and defined here before it appears below):

Value: one cell of data.
Record, one row: a list of cells bound to a schema.
Schema: the column names and their order.
Arc, a shared handle, so every row points at the same schema.

Predict first

✋ Predict first

A CSV file has the header line `customer_id,first_name,lifetime_value` and then a data row `1001,Alice,15200`. The CSV source reads that row into a record. The cell holding 15200: what is its TYPE inside the record right after reading: an integer, a float, or a string? Write your answer and one sentence of why before revealing.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

It’s a string, Value::String("15200"), not an integer.

The CSV reader does not guess types. A .csv file is just text; 15200 and Alice arrive identically, as runs of characters. So the reader reads every field as a string and stops there. Turning "15200" into the number 15200 is a later step, a transform’s job, not the reader’s. Right after reading, Alice’s whole row is a row of strings.

The trap is assuming the reader is “smart” and infers 15200 is a number because it looks like one. It doesn’t, and that’s deliberate: type inference at read time would make two files with the same data parse into different types depending on their contents. Clinker keeps reading dumb and predictable, then coerces on purpose later. Hold that: the worked example below builds exactly this all-strings row.

A cell is a `Value`

Every single cell of data in Clinker is one Value, a closed set of nine shapes (null, bool, integer, float, string, date, datetime, array, map):

clinker-record ·value.rs ·Value type @19acdcb4

pub enum Value {
    Null,
    Bool(bool),
    Integer(i64),
    Float(f64),
    String(FieldStr),
    Date(NaiveDate),
    DateTime(NaiveDateTime),
    Array(Vec<Value>),
    Map(Box<IndexMap<Box<str>, Value>>),
}

It’s a closed set: a cell is exactly one of these nine, never something else. When the CSV source reads Alice’s row, it doesn’t guess types; it reads every field as a string (Value::String), exactly as you predicted. So right after reading, Alice is a row of strings; coercing "15200" to a number is a later transform’s job.

A row is a `Record`

A Record is one row: a list of Values lined up against a Schema that names the columns.

clinker-record ·mod.rs ·Record type @19acdcb4

The real definition, trimmed to the part that matters now:

pub struct Record {
    schema: Arc<Schema>,   // the column names + order, shared by every row of a source
    values: Vec<Value>,    // the cells, positional — values[i] belongs to column i
    // ... plus per-record scoped vars and document context, for later
}

Two ideas to take away. First, the cells are a plain Vec<Value> indexed by position; there’s no name stored next to each cell. Second, the schema is held behind an Arc, a shared handle, so a million rows from one source all point at the same schema instead of each carrying their own copy. (Why Arc, and what that costs, is a Data & Representation question.)

The columns are a `Schema`

The Schema is the list of column names and their order; it’s what lets values[5] mean “lifetime_value”:

clinker-record ·schema.rs ·Schema type @19acdcb4

pub struct Schema {
    columns: Vec<Box<str>>,           // column names, in order
    field_metadata: Vec<Option<FieldMetadata>>,
    index: HashMap<Box<str>, usize>,  // name -> position, for O(1) lookup
}

This is the join that makes a positional Vec<Value> readable. To find “lifetime_value”, the schema’s index maps the name to a position (say 5); then values[5] is the cell. The names live once, in the schema; the cells are bare values keyed by their slot. In customer_etl, the source declared the schema right in the YAML: customer_id, first_name, last_name, email, status, lifetime_value, zip_code. That’s the schema every customer record is bound to.

Worked → completion → faded

Now build a record yourself, scaffolded down from a fully worked model to one you write from scratch. Each rung adds exactly one new idea.

Worked: a record, modeled in miniature

Here’s Alice’s row as the reader first sees it: column names plus a Vec of values, every cell a string. Run it and read the output, each column name printed against its positional cell.

rust // editable

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String), // the real engine uses an optimized string type — see the FieldStr lesson
}

fn main() {
  // A schema: column names, in order. The names live here, once.
  let schema = ["customer_id", "first_name", "status", "lifetime_value"];

  // A record: positional values, one per column. The CSV reader produces
  // strings — coercion to numbers happens later, in a transform.
  let alice: Vec<Value> = vec![
      Value::Str("1001".to_string()),
      Value::Str("Alice".to_string()),
      Value::Str("active".to_string()),
      Value::Str("15200".to_string()),
  ];

  // zip(schema, alice) re-pairs each name with its positional cell.
  for (column, value) in schema.iter().zip(&alice) {
      println!("{column:>15} = {value:?}");
  }
}

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String), // the real engine uses an optimized string type — see the FieldStr lesson
}

fn main() {
  // A schema: column names, in order. The names live here, once.
  let schema = ["customer_id", "first_name", "status", "lifetime_value"];

  // A record: positional values, one per column. The CSV reader produces
  // strings — coercion to numbers happens later, in a transform.
  let alice: Vec<Value> = vec![
      Value::Str("1001".to_string()),
      Value::Str("Alice".to_string()),
      Value::Str("active".to_string()),
      Value::Str("15200".to_string()),
  ];

  // zip(schema, alice) re-pairs each name with its positional cell.
  for (column, value) in schema.iter().zip(&alice) {
      println!("{column:>15} = {value:?}");
  }
}

> output appears here — press Run

The zip is the whole point in miniature: the names and the cells are stored apart (one schema, one Vec) and re-joined by position when you need them, exactly how the real Record uses Arc<Schema> + Vec<Value>.

Completion: add the missing column

Below, the same record is missing its last column. The schema already lists zip_code, but alice has only four cells, so the zip stops short and zip_code never prints. Add the fifth cell so every column has a value. (Like the CSV reader, make it a Value::Str, not an integer.)

rust // editable

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String),
}

fn main() {
  let schema = ["customer_id", "first_name", "status", "lifetime_value", "zip_code"];

  let alice: Vec<Value> = vec![
      Value::Str("1001".to_string()),
      Value::Str("Alice".to_string()),
      Value::Str("active".to_string()),
      Value::Str("15200".to_string()),
      // <- add the zip_code cell here so the Vec lines up with the schema
  ];

  for (column, value) in schema.iter().zip(&alice) {
      println!("{column:>15} = {value:?}");
  }
}

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String),
}

fn main() {
  let schema = ["customer_id", "first_name", "status", "lifetime_value", "zip_code"];

  let alice: Vec<Value> = vec![
      Value::Str("1001".to_string()),
      Value::Str("Alice".to_string()),
      Value::Str("active".to_string()),
      Value::Str("15200".to_string()),
      // <- add the zip_code cell here so the Vec lines up with the schema
  ];

  for (column, value) in schema.iter().zip(&alice) {
      println!("{column:>15} = {value:?}");
  }
}

> output appears here — press Run

💡 Hint 1

zip pairs by position and stops at the shorter side. The schema has five names; alice has four cells, so zip_code is dropped. Push one more Value::Str(...) onto the Vec: a zip code like "94103", read as a string, just like every other freshly-read cell.

Show solution

let alice: Vec<Value> = vec![
    Value::Str("1001".to_string()),
    Value::Str("Alice".to_string()),
    Value::Str("active".to_string()),
    Value::Str("15200".to_string()),
    Value::Str("94103".to_string()), // zip_code — a string, like every read cell
];

A record’s Vec<Value> must have exactly one cell per schema column, in order. The zip code is text on disk, so it arrives as Value::Str, not Value::Integer, even though it looks numeric. The schema declares which columns exist; the Vec supplies one positional cell for each.

Faded: build the record from scratch

Your turn with much less scaffolding. The schema is given. Build the Vec<Value> for Bob’s row (1002, Bob, active, 8400, 10001) as the CSV reader would first produce it (every cell a string), then print each column against its cell with the same zip loop. The Value enum and main skeleton are provided; you write the Vec and the loop.

rust // editable

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String),
}

fn main() {
  let schema = ["customer_id", "first_name", "status", "lifetime_value", "zip_code"];

  // TODO: build Bob's row as the reader sees it — five string cells,
  // one per schema column, in order: "1002", "Bob", "active", "8400", "10001".
  let bob: Vec<Value> = todo!();

  // TODO: pair each column name with its positional cell and print them,
  // the way the worked example did.
}

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String),
}

fn main() {
  let schema = ["customer_id", "first_name", "status", "lifetime_value", "zip_code"];

  // TODO: build Bob's row as the reader sees it — five string cells,
  // one per schema column, in order: "1002", "Bob", "active", "8400", "10001".
  let bob: Vec<Value> = todo!();

  // TODO: pair each column name with its positional cell and print them,
  // the way the worked example did.
}

> output appears here — press Run

Show solution

fn main() {
    let schema = ["customer_id", "first_name", "status", "lifetime_value", "zip_code"];

    let bob: Vec<Value> = vec![
        Value::Str("1002".to_string()),
        Value::Str("Bob".to_string()),
        Value::Str("active".to_string()),
        Value::Str("8400".to_string()),
        Value::Str("10001".to_string()),
    ];

    for (column, value) in schema.iter().zip(&bob) {
        println!("{column:>15} = {value:?}");
    }
}

Five columns, five string cells, joined by position. Note that lifetime_value is "8400" (a string) here, not 8400 (an integer); coercion is the next transform’s job, not the reader’s. You just built, by hand, the exact shape the CSV source hands the rest of the pipeline.

Why-bridge: why a positional `Vec` and a shared `Arc<Schema>`?

Two design choices in the real Record are worth a sentence of why, since you’ll lean on them all through Data & Representation.

Cells are positional, not named per cell. Storing the column name next to every cell would repeat the same seven strings on every one of a million rows. Instead the names live once in the Schema, and a cell is found by its slot: values[i]. The cost is that a bare cell is meaningless without its schema, which is why a Record always carries one.
The schema is shared behind Arc, not copied. Every row of one source has the same columns, so copying the schema per row would be pure waste. Arc lets a million records share one schema: cloning a record’s schema handle bumps a reference count, it doesn’t duplicate the column list.

Retrieval checkpoint

✓ Checkpoint

// quick check

Right after the CSV source reads Alice's row, what is the type of her `lifetime_value` cell?

The reader makes every field a Value::String. Converting the string 15200 to a number is a transform's job (you'll watch it happen when you trace a record end-to-end), not the reader's.

// quick check

A Record holds its cells as a Vec<Value>. How does it know which cell belongs to which column?

Cells are positional: values[i] belongs to the i-th column in the Arc. The schema is stored once and shared by every record of the source, not copied per row, and the name is not repeated on each cell.

// quick check

Two engineers describe the Arc<Schema> on a Record. Which statement is the MISCONCEPTION to correct?

Misconception: records do NOT each carry a private copy. Arc means every row of a source points at the same shared schema; that's the whole reason the column names aren't repeated per record. (A second common misconception: that a freshly-read cell is already typed. It's a Value::String until a transform coerces it.)

// quick check

Spaced recall from Orientation: you ran `customer_etl` with `--dry-run` and saw `5 total, 5 ok, 5 written, 0 dlq`. What does each of those 5 records correspond to, in the vocabulary of this lesson?

Spaced recall from Orientation: the total count in the --dry-run summary counts records, the rows. Each is a Record: a positional Vec against one shared Arc. The schema is declared once for the source, not rebuilt per row, and a cell (Value) is one column of one record, not the whole record.

You should be able to:

You can name the three types: Value (a cell), Record (a row), Schema (the columns)
You can explain why a freshly-read CSV record is all Value::String
You can say how values[i] finds the right column (positional, via the shared Schema)
You built a record by hand (a schema plus a positional Vec of string cells) and printed each cell against its column
You read the real Record struct and found the Vec<Value> + Arc<Schema>

Verify in the checkout:

grep -nA4 'pub struct Record' crates/clinker-record/src/record/mod.rs
grep -n 'pub enum Value' crates/clinker-record/src/value.rs
grep -nA4 'pub struct Schema' crates/clinker-record/src/schema.rs

Connections

You now have the vocabulary of a record: a Value is a cell, a Record is a positional row of cells, and a Schema names the columns once and is shared via Arc. Next: how the YAML pipeline you wrote becomes a plan the engine can actually run.

See it from the pipeline author’s side (optional, one-directional; the schema and record concept where you declare the schema: in YAML rather than read the Rust types):

Glossary terms used: Value, Record, Schema, Arc.

What is a record?

What you’ll be able to do

Predict first

A cell is a Value

A row is a Record

The columns are a Schema

Worked → completion → faded

Worked: a record, modeled in miniature

Completion: add the missing column

Faded: build the record from scratch

Why-bridge: why a positional Vec and a shared Arc<Schema>?

Retrieval checkpoint

Connections

A cell is a `Value`

A row is a `Record`

The columns are a `Schema`

Why-bridge: why a positional `Vec` and a shared `Arc<Schema>`?