Skip to content
Clinker Academy

Provenance — where a value came from

Every record in Clinker drags along a little record of where it came from, its provenance, and that is what lets a failure report “row 42 of customers.csv” instead of “a record failed.” Provenance is several facts that only mean something together: a file, a row number, a batch, a timestamp. You spent the last few lessons inside enum, a value that is exactly one of its shapes; provenance is the opposite shape, all of its facts at once. The Rust tool for “all of these fields, together” is the struct; we use just enough of it to read the real RecordProvenance and build a miniature, and The Rust Book, ch. 5 is the canonical treatment if you want structs from first principles.

What you’ll be able to do

Section titled “What you’ll be able to do”
  • Define a struct with named fields and hang a method off it in an impl block.
  • Distinguish a struct (a bundle of all its fields, an AND) from an enum (exactly one variant, an OR).
  • Explain why every record carries a provenance struct, and what a dead-letter entry gains from it.
  • Predict what cloning a provenance does to the shared filename behind its Arc<str>: a counter bump, not a string copy.

New terms in this lesson (each is also expanded inline at the point you first need it):

  • struct: a named bundle of fields that belong together; all of them at once.
  • field: one named, typed component of a struct, reached with .name.
  • provenance: the origin facts a record carries: which file, which row, which batch, ingested when.
  • derive: #[derive(...)], asking the compiler to write a trait impl (like Debug) for you.
  • Arc: shared ownership of one heap value; cloning copies the handle, not the data.

Predict first

Section titled “Predict first”

struct: AND, where enum is OR

Section titled “struct: AND, where enum is OR”

You spent the last few lessons inside enum, a value that is exactly one of its variants. A struct is the other half of the pair. Where an enum value is one variant or another, a struct value is all of its fields and each other, at once:

struct Point { x: i64, y: i64 }   // a Point has an x AND a y

Each named piece is a field, reached with dot syntax (p.x). An impl block hangs methods off the struct: functions that take &self and read those fields. Records, schemas, plans, the engine’s larger types are all structs, because each is several facts that always travel as a unit.

Provenance: every row remembers its origin

Section titled “Provenance: every row remembers its origin”

When a record fails, “row 42 of customers.csv” is far more useful than “a record failed.” So every record carries a small struct describing where it came from, its provenance:

clinker-record ·provenance.rs ·RecordProvenance type @19acdcb4 
pub struct RecordProvenance {
    pub source_file: Arc<str>,      // the file this row came from (shared)
    pub source_row: u64,            // its position in that file
    pub source_batch: Arc<str>,     // the ingestion batch (also shared)
    pub ingestion_timestamp: NaiveDateTime,
}

This is what lets a dead-letter entry point at the exact source row, and what makes error messages specific. It’s a struct precisely because origin is several facts bundled together (a file, a position, a batch, a time) that always travel as a unit. A Null is one or the other shape; a provenance is all four of these facts at once. That “all of them, together” is exactly what a struct is for.

Sharing the origin cheaply

Section titled “Sharing the origin cheaply”

Here’s the engineering subtlety, and the one you predicted at the top. A million rows from customers.csv all share the same filename. Storing the string "customers.csv" a million times would be wasteful, so the filename is held behind an Arc<str>, a shared, reference-counted handle, and every record of that file points at the same one.

You met Arc and .clone() last in the ownership lesson: a plain .clone() on a String deep-copies its bytes, but Arc::clone copies only the handle and bumps a counter. So cloning the provenance for a new row doesn’t copy the filename; it shares it. The real engine even has a factory that captures the shared file and batch once, then stamps each row by cloning those handles:

// from RecordProvenance — captures shared handles once, stamps each row cheaply
pub fn factory(source_file: Arc<str>, source_batch: Arc<str>, ts: NaiveDateTime)
    -> impl Fn(u64) -> RecordProvenance
{
    move |source_row| RecordProvenance {
        source_file: Arc::clone(&source_file),    // share the handle — no string copy
        source_row,                               // the only per-row fact that differs
        source_batch: Arc::clone(&source_batch),  // share the handle again
        ingestion_timestamp: ts,
    }
}

(That Arc sharing pattern is the whole of the smart-pointers lesson. Here you only need to read it: clone the handle, not the bytes.)

Worked → completion → faded

Section titled “Worked → completion → faded”

You’ve read the real provenance struct. Now you build the machinery yourself, scaffolded down to an unaided task. One new idea per rung: a struct with a method, then a method that reads more fields, then a from-scratch struct whose clone shares a handle.

Worked: a struct with a method

Section titled “Worked: a struct with a method”

Here is a small, self-contained provenance struct with one method. Run it as given. The #[derive(Debug)] line is a derive: it asks the compiler to write the Debug impl so {p:?} can print the struct.

rust // editable

The struct is one named thing holding file AND row. The method borrows it with &self (you saw & is a borrow back in move & borrow), so calling describe() reads the fields without consuming the struct, and p is still usable afterward.

Completion: read one more field

Section titled “Completion: read one more field”

Below, Provenance gains a batch field, and describe is half-written. Complete the format! so the output also names the batch; read self.batch the same way the other fields are read. The skeleton compiles once you fill the one gap.

rust // editable
💡 Hint 1
The pattern is the same as the other fields: a {} placeholder in the string, then self.batch added to the argument list. Read it off self exactly like self.row and self.file.
Show solution 
fn describe(&self) -> String {
    format!("row {} of {} (batch {})", self.row, self.file, self.batch)
}

Every value of the struct has every field, so self.batch is always there to read. That’s the AND nature of a struct doing its job. Add a field and every method can reach it through self; the struct guarantees it’s present.

Faded: share the filename behind an Arc

Section titled “Faded: share the filename behind an Arc”

Now with much less scaffolding, and the real engine’s trick. Write a Provenance whose file is an Arc<str> (the shared handle), give it a new constructor, then in main build two provenances for different rows of the same file by cloning that one shared handle, and prove they point at the same allocation with Arc::ptr_eq.

rust // editable
Show solution 
use std::sync::Arc;


#[derive(Debug)]
struct Provenance {
    file: Arc<str>,
    row: u64,
}


impl Provenance {
    fn new(file: Arc<str>, row: u64) -> Self {
        Provenance { file, row } // field-init shorthand: `file: file` written once
    }
}


fn main() {
    let file: Arc<str> = Arc::from("customers.csv"); // ONE allocation


    let p1 = Provenance::new(Arc::clone(&file), 41); // share the handle — no copy
    let p2 = Provenance::new(Arc::clone(&file), 42); // share it again


    println!("{p1:?}");
    println!("{p2:?}");


    // Both point at the SAME string in memory:
    assert!(Arc::ptr_eq(&p1.file, &p2.file));
    println!("same filename allocation: {}", Arc::ptr_eq(&p1.file, &p2.file));
}

Arc::clone(&file) is the whole move: each new call shares the one filename allocation rather than copying it. Arc::ptr_eq returns true because both handles point at the same bytes, exactly what the real RecordProvenance::factory does for every row of a file. The row is the only per-row fact that actually differs.

Why-bridge: a struct so errors can name the source

Section titled “Why-bridge: a struct so errors can name the source”

Why does every record drag a provenance struct along? Because a failure has to be locatable. When a coercion fails on row 4 million, the engine routes that record to the dead-letter queue, and a DLQ entry is only useful if it can say which row of which file failed. That information has to ride with the record itself, all the way from the reader to the error path, or it’s gone by the time the failure surfaces.

A struct is the right shape for it because origin is several facts that are meaningless apart: a row number with no file, or a file with no row, locates nothing. Bundling them guarantees they travel together. And the Arc<str> on the shared fields is what makes carrying provenance on every record affordable: one filename allocation per file, shared by a counter bump, instead of a fresh copy per row.

Retrieval checkpoint

Section titled “Retrieval checkpoint”

Connections

Section titled “Connections”

You can bundle data into structs, hang methods off them with impl, and you’ve seen the Arc<str> sharing trick that makes carrying provenance on every record affordable. Next: the collections that make up a record, and the iterators that stream them one at a time.

Go deeper on the Rust (optional, one-directional, the same Arc sharing taught from first principles, no engine framing required):

Glossary terms used: struct, field, provenance, derive, Arc.