What is a record?

Back in Orientation you ran customer_etl and Clinker reported “5 records.” A Record’s Journey follows one of those records all the way through the engine. But first: what is a record? It’s the unit of data that flows through every pipeline, and it’s built from three vocabulary types you’ll see everywhere: a cell, a row, and the columns that name the cells.

What you’ll be able to do

Name the three types that make up a record (Value is a cell, Record a row, Schema the columns) and say what each one holds.
Explain how the declared schema turns decoded CSV text into typed values before downstream processing.
Trace how a cell is located: how values[i] finds the right column through the shared Schema.
Write a miniature record (a schema plus a positional Vec of values) and print each cell against its column name.

(We meet these types shallowly here; Data & Representation opens each one up: the optimized string type, the Arc sharing, the per-record context.)

New terms in this lesson (each is named and defined here before it appears below):

Value: one cell of data.
Record, one row: a list of cells bound to a schema.
Schema: the column names and their order.
Arc, a shared handle, so every row points at the same schema.

Predict first

✋ Predict first

A CSV cell contains 15200. The source schema declares lifetime_value as int. What value reaches the next Transform: text or an integer? What changes if the schema says string?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

With type: int, source admission produces Value::Integer(15200) before the record reaches the Transform. With type: string, it preserves the text as Value::String. The CSV decoder itself reads text cells, but it is only the first part of ingestion: the declared-schema boundary applies immediately after decoding. It does not guess a type from the cell’s appearance.

The worked customer example below deliberately declares all its columns as strings. That choice explains its all-text input row and later explicit .to_int() call; it is not a restriction of CSV pipelines. A value that cannot satisfy its declared source type rejects the row under the error policy.

A cell is a `Value`

Every single cell of data in Clinker is one Value, a closed set of ten shapes (null, bool, integer, float, decimal, string, date, datetime, array, map):

clinker-record ·value.rs ·Value type @3b343a4e

pub enum Value {
    Null,
    Bool(bool),
    Integer(i64),
    Float(f64),
    Decimal(Decimal),
    String(FieldStr),
    Date(NaiveDate),
    DateTime(NaiveDateTime),
    Array(OwnedValues),
    Map(OwnedMap),
}

It’s a closed set: a cell is exactly one of these ten, never something else. The CSV decoder produces text, and source admission applies the declared schema. Alice reaches the next node as a row of strings because this example deliberately declares string columns. A declared numeric column would already hold a parsed number at that boundary.

A row is a `Record`

A Record is one row: a list of Values lined up against a Schema that names the columns.

clinker-record ·mod.rs ·Record type @3b343a4e

The real definition, trimmed to the part that matters now:

pub struct Record {
    schema: SharedStorage<Schema>,   // the column names + order, shared by every row of a source
    values: OwnedValues,    // the cells, positional — values[i] belongs to column i
    // ... plus per-record scoped vars and document context, for later
}

Two ideas to take away. First, the cells are a plain OwnedValues indexed by position; there’s no name stored next to each cell. Second, the schema is held behind an Arc, a shared handle, so a million rows from one source all point at the same schema instead of each carrying their own copy. (Why Arc, and what that costs, is a Data & Representation question.)

The columns are a `Schema`

The Schema is the list of column names and their order; it’s what lets values[5] mean “lifetime_value”:

clinker-record ·schema.rs ·Schema type @3b343a4e

pub struct Schema {
    columns: Vec<Box<str>>,           // column names, in order
    field_metadata: Vec<Option<FieldMetadata>>,
    index: HashMap<Box<str>, usize>,  // name -> position, for O(1) lookup
}

This is the join that makes a positional OwnedValues readable. To find “lifetime_value”, the schema’s index maps the name to a position (say 5); then values[5] is the cell. The names live once, in the schema; the cells are bare values keyed by their slot. In customer_etl, the source declared the schema right in the YAML: customer_id, first_name, last_name, email, status, lifetime_value, zip_code. That’s the schema every customer record is bound to.

Worked → completion → faded

Now build a record yourself, scaffolded down from a fully worked model to one you write from scratch. Each rung adds exactly one new idea.

Worked: a record, modeled in miniature

Here’s Alice’s row as the reader first sees it: column names plus a Vec of values, every cell a string. Run it and read the output, each column name printed against its positional cell.

rust // editable

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String), // the real engine uses an optimized string type — see the FieldStr lesson
}

fn main() {
  // A schema: column names, in order. The names live here, once.
  let schema = ["customer_id", "first_name", "status", "lifetime_value"];

  // A record: positional values, one per column. The CSV reader produces
  // strings because this example deliberately declares string columns.
  let alice: Vec<Value> = vec![
      Value::Str("1001".to_string()),
      Value::Str("Alice".to_string()),
      Value::Str("active".to_string()),
      Value::Str("15200".to_string()),
  ];

  // zip(schema, alice) re-pairs each name with its positional cell.
  for (column, value) in schema.iter().zip(&alice) {
      println!("{column:>15} = {value:?}");
  }
}

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String), // the real engine uses an optimized string type — see the FieldStr lesson
}

fn main() {
  // A schema: column names, in order. The names live here, once.
  let schema = ["customer_id", "first_name", "status", "lifetime_value"];

  // A record: positional values, one per column. The CSV reader produces
  // strings because this example deliberately declares string columns.
  let alice: Vec<Value> = vec![
      Value::Str("1001".to_string()),
      Value::Str("Alice".to_string()),
      Value::Str("active".to_string()),
      Value::Str("15200".to_string()),
  ];

  // zip(schema, alice) re-pairs each name with its positional cell.
  for (column, value) in schema.iter().zip(&alice) {
      println!("{column:>15} = {value:?}");
  }
}

> output appears here — press Run

The zip is the whole point in miniature: the names and the cells are stored apart (one schema, one Vec) and re-joined by position when you need them, exactly how the real Record uses SharedStorage<Schema> + OwnedValues.

Completion: add the missing column

Below, the same record is missing its last column. The schema already lists zip_code, but alice has only four cells, so the zip stops short and zip_code never prints. Add the fifth cell so every column has a value. (Like the CSV reader, make it a Value::Str, not an integer.)

rust // editable

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String),
}

fn main() {
  let schema = ["customer_id", "first_name", "status", "lifetime_value", "zip_code"];

  let alice: Vec<Value> = vec![
      Value::Str("1001".to_string()),
      Value::Str("Alice".to_string()),
      Value::Str("active".to_string()),
      Value::Str("15200".to_string()),
      // <- add the zip_code cell here so the Vec lines up with the schema
  ];

  for (column, value) in schema.iter().zip(&alice) {
      println!("{column:>15} = {value:?}");
  }
}

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String),
}

fn main() {
  let schema = ["customer_id", "first_name", "status", "lifetime_value", "zip_code"];

  let alice: Vec<Value> = vec![
      Value::Str("1001".to_string()),
      Value::Str("Alice".to_string()),
      Value::Str("active".to_string()),
      Value::Str("15200".to_string()),
      // <- add the zip_code cell here so the Vec lines up with the schema
  ];

  for (column, value) in schema.iter().zip(&alice) {
      println!("{column:>15} = {value:?}");
  }
}

> output appears here — press Run

💡 Hint 1

zip pairs by position and stops at the shorter side. The schema has five names; alice has four cells, so zip_code is dropped. Push one more Value::Str(...) onto the Vec: a zip code like "94103", read as a string, just like every other freshly-read cell.

Show solution

let alice: OwnedValues = vec![
    Value::Str("1001".to_string()),
    Value::Str("Alice".to_string()),
    Value::Str("active".to_string()),
    Value::Str("15200".to_string()),
    Value::Str("94103".to_string()), // zip_code — a string, like every read cell
];

A record’s OwnedValues must have exactly one cell per schema column, in order. The zip code is text on disk, so it arrives as Value::Str, not Value::Integer, even though it looks numeric. The schema declares which columns exist; the Vec supplies one positional cell for each.

Faded: build the record from scratch

Your turn with much less scaffolding. The schema is given. Build the OwnedValues for Bob’s row (1002, Bob, active, 8400, 10001) as the CSV reader would first produce it (every cell a string), then print each column against its cell with the same zip loop. The Value enum and main skeleton are provided; you write the Vec and the loop.

rust // editable

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String),
}

fn main() {
  let schema = ["customer_id", "first_name", "status", "lifetime_value", "zip_code"];

  // TODO: build Bob's row as the reader sees it — five string cells,
  // one per schema column, in order: "1002", "Bob", "active", "8400", "10001".
  let bob: Vec<Value> = todo!();

  // TODO: pair each column name with its positional cell and print them,
  // the way the worked example did.
}

#[derive(Debug)]
enum Value {
  Null,
  Integer(i64),
  Str(String),
}

fn main() {
  let schema = ["customer_id", "first_name", "status", "lifetime_value", "zip_code"];

  // TODO: build Bob's row as the reader sees it — five string cells,
  // one per schema column, in order: "1002", "Bob", "active", "8400", "10001".
  let bob: Vec<Value> = todo!();

  // TODO: pair each column name with its positional cell and print them,
  // the way the worked example did.
}

> output appears here — press Run

Show solution

fn main() {
    let schema = ["customer_id", "first_name", "status", "lifetime_value", "zip_code"];

    let bob: OwnedValues = vec![
        Value::Str("1002".to_string()),
        Value::Str("Bob".to_string()),
        Value::Str("active".to_string()),
        Value::Str("8400".to_string()),
        Value::Str("10001".to_string()),
    ];

    for (column, value) in schema.iter().zip(&bob) {
        println!("{column:>15} = {value:?}");
    }
}

Five columns, five string cells, joined by position. Note that lifetime_value is "8400" (a string) here, not 8400 (an integer); coercion is the next transform’s job, not the reader’s. You just built, by hand, the exact shape the CSV source hands the rest of the pipeline.

Why-bridge: why a positional `Vec` and a shared `SharedStorage<Schema>`?

Two design choices in the real Record are worth a sentence of why, since you’ll lean on them all through Data & Representation.

Cells are positional, not named per cell. Storing the column name next to every cell would repeat the same seven strings on every one of a million rows. Instead the names live once in the Schema, and a cell is found by its slot: values[i]. The cost is that a bare cell is meaningless without its schema, which is why a Record always carries one.
The schema is shared behind Arc, not copied. Every row of one source has the same columns, so copying the schema per row would be pure waste. Arc lets a million records share one schema: cloning a record’s schema handle bumps a reference count, it doesn’t duplicate the column list.

Retrieval checkpoint

✓ Checkpoint

// quick check

Right after the CSV source reads Alice's row, what is the type of her `lifetime_value` cell?

This example declares lifetime_value as string, so source admission preserves Value::String. Declaring it as int would parse 15200 into Value::Integer before downstream nodes; an explicit transform cast is only needed when the declared schema keeps it as string.

// quick check

A Record holds its cells as an OwnedValues. How does it know which cell belongs to which column?

Cells are positional: values[i] belongs to the i-th column in the Arc. The schema is stored once and shared by every record of the source, not copied per row, and the name is not repeated on each cell.

// quick check

Two engineers describe the SharedStorage<Schema> on a Record. Which statement is the MISCONCEPTION to correct?

Misconception: records do NOT each carry a private copy. Arc means every row of a source points at the same shared schema; that's the whole reason the column names aren't repeated per record. (A second common misconception: that a freshly-read cell is already typed. It's a Value::String until a transform coerces it.)

// quick check

Spaced recall from Orientation: you ran `customer_etl` with `--dry-run` and saw `5 total, 5 ok, 5 written, 0 dlq`. What does each of those 5 records correspond to, in the vocabulary of this lesson?

Spaced recall from Orientation: the total count in the --dry-run summary counts records, the rows. Each is a Record: a positional Vec against one shared Arc. The schema is declared once for the source, not rebuilt per row, and a cell (Value) is one column of one record, not the whole record.

You should be able to:

You can name the three types: Value (a cell), Record (a row), Schema (the columns)
You can explain why a freshly-read CSV record is all Value::String
You can say how values[i] finds the right column (positional, via the shared Schema)
You built a record by hand (a schema plus a positional Vec of string cells) and printed each cell against its column
You read the real Record struct and found the OwnedValues + SharedStorage<Schema>

Verify in the checkout:

grep -nA4 'pub struct Record' crates/clinker-record/src/record/mod.rs
grep -n 'pub enum Value' crates/clinker-record/src/value.rs
grep -nA4 'pub struct Schema' crates/clinker-record/src/schema.rs

Connections

You now have the vocabulary of a record: a Value is a cell, a Record is a positional row of cells, and a Schema names the columns once and is shared via Arc. Next: how the YAML pipeline you wrote becomes a plan the engine can actually run.

See it from the pipeline author’s side (optional, one-directional; the schema and record concept where you declare the schema: in YAML rather than read the Rust types):

Glossary terms used: Value, Record, Schema, Arc.

What is a record?

What you’ll be able to do

Predict first

A cell is a Value

A row is a Record

The columns are a Schema

Worked → completion → faded

Worked: a record, modeled in miniature

Completion: add the missing column

Faded: build the record from scratch

Why-bridge: why a positional Vec and a shared SharedStorage<Schema>?

Retrieval checkpoint

Connections

A cell is a `Value`

A row is a `Record`

The columns are a `Schema`

Why-bridge: why a positional `Vec` and a shared `SharedStorage<Schema>`?