Sharing the schema, boxing the map — smart pointers

The engine shares one Schema across every record of a file instead of copying it a million times, and it boxes the map inside Value so a single cell stays small. These are two everyday moves that decide how much memory a run costs. You’ve already seen the sharing half three times without naming the tool: an Arc<Schema> while streaming records, an Arc<str> filename in provenance, a shared document context. This lesson makes the tools explicit: Box (put one thing on the heap), Rc and Arc (let many owners share one thing). It also shows why one choice, Arc over Rc, is forced by how Clinker runs. The Rust tools underneath are the smart pointers; we use just enough to read the engine’s types, and The Rust Book, ch. 15 is the canonical treatment if you want them from first principles.

Prerequisites

• Streaming records

→

Leads to

• Zero-copy reads

What you’ll be able to do

• Name what Box, Rc, and Arc each give you: one heap owner, many single-threaded owners, many cross-thread owners.
• Explain why cloning an Arc<Schema> is O(1): a pointer copy plus a reference-count bump, not a copy of the schema.
• Distinguish Rc from Arc by their reference count (plain integer vs atomic) and say which one a thread-crossing value must use.
• Write a small program that shares one heap value across several owners and reads the live reference count.

New terms in this lesson (each is also expanded inline at the point you first need it):

• heap: the run-time pool you allocate from, reached through a pointer.
• Box<T>: an owning pointer to one heap value; a single owner.
• Arc<T>: shared ownership of one heap value across many owners and threads.
• reference count: the integer counting how many owning handles exist.
• shared ownership: one value with several genuine, independent owners at once.

Predict first

✋ Predict first

A schema is built once, then handed to a million records that each store an Arc<Schema>. How much schema data does the engine copy when it makes the millionth handle: the whole schema, half of it, or none of it? Write your answer before revealing.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

None of it. Arc::clone copies a pointer and bumps a reference count, an O(1) operation regardless of how big the schema is. There is exactly one Schema allocation on the heap; the million records all hold cheap owning handles to it. The schema is freed once, when the last of those handles drops.

The trap is reading the word “clone” as “copy.” Arc::clone is named clone because it satisfies the Clone trait, but it deliberately does not duplicate the pointed-to value the way Vec::clone or String::clone does; it duplicates the handle. Hold that distinction; the cost difference between “clone the handle” and “clone the data” is the whole reason the engine can afford to share a schema a million ways.

Box: one owner, on the heap

Start with the simplest smart pointer. A Box<T> puts a T on the heap and owns it; the box itself is just an 8-byte pointer on the stack. You already met its main job in The Value cell: Map(Box<...>) boxes the map so the Value enum stays small; the variant holds a pointer, not the whole map. Box is also what makes recursive types possible (a Value can contain Values): without the indirection, the type would be infinitely sized.

Box is single-owner, like an ordinary owned value: when the box drops, the heap value is freed. What it does not give you is sharing. For that you need a reference-counted pointer.

Rc vs Arc: many owners, one allocation

Sometimes many things need to share the same data, like every record of a file sharing one Schema. That’s what Rc and Arc do: they give you shared ownership. They wrap a value in a reference count, hand out cheap clones (each clone only bumps the count), and free the value when the last owner drops. Cloning an Arc<Schema> doesn’t copy the schema; it copies a pointer and adds one.

The difference between them is thread safety. Rc’s counter is a plain integer: fast, but unsafe to touch from two threads. Arc’s counter is atomic, safe across threads at a tiny cost. Clinker reads each source on its own thread and runs heavy operators on a thread pool, so records and the things they share must be able to cross threads. That’s why the engine uses Arc everywhere it shares; an Rc would not even compile in a value that crosses a thread boundary.

You can see it in the per-document context that records share:

clinker-record ·document_context.rs ·DocumentContext type @19acdcb4

pub struct DocumentContext {
    id: DocumentId,
    grain: DocumentGrain,
    source_file: Arc<str>,   // shared per source file — every record points here
    // ...
}

and in the string storage behind Value::String, which uses an Arc to share long strings across clones (its full design, three storage strategies in one type, is an Execution & Memory deep dive):

clinker-record ·field_str.rs ·FieldStr type @19acdcb4

/// A field-value string stored inline, `Arc`-shared, or `Box`-unique
/// behind a single `str` API. 24 bytes wide.
pub struct FieldStr {
    repr: Repr,   // inline bytes, an Arc<str> (shared), or a Box<str> (unique)
}

Notice that FieldStr reaches for both smart pointers, for the two different jobs you just met: Box<str> when one owner holds a unique string, Arc<str> when the same string is shared across clones.

Worked → completion → faded

Now you build the share-without-copying pattern yourself, scaffolded down to your own code. One new idea per rung: read the count, then change an owner set, then write the whole thing.

Worked: share one allocation, read the count

Here is a self-contained program: one Schema on the heap, three owning handles, and the live reference count printed. Run it and watch the count rise.

rust // editable

reset ▶ run

use std::sync::Arc;

#[derive(Debug)]
struct Schema { columns: Vec<String> }

fn main() {
  // One schema, allocated ONCE on the heap, owned through an Arc.
  let schema = Arc::new(Schema { columns: vec!["id".into(), "status".into()] });

  // Each Arc::clone is a pointer copy + a refcount bump — NOT a copy of the schema.
  let r1 = Arc::clone(&schema);
  let r2 = Arc::clone(&schema);

  // strong_count reports how many owning handles currently exist.
  println!("shared by {} owners", Arc::strong_count(&schema)); // 3
  println!("r1 sees columns: {:?}", r1.columns);
  println!("r2 sees the same schema: {:?}", r2.columns);
}

use std::sync::Arc; #[derive(Debug)] struct Schema { columns: Vec<String> } fn main() { // One schema, allocated ONCE on the heap, owned through an Arc. let schema = Arc::new(Schema { columns: vec!["id".into(), "status".into()] }); // Each Arc::clone is a pointer copy + a refcount bump — NOT a copy of the schema. let r1 = Arc::clone(&schema); let r2 = Arc::clone(&schema); // strong_count reports how many owning handles currently exist. println!("shared by {} owners", Arc::strong_count(&schema)); // 3 println!("r1 sees columns: {:?}", r1.columns); println!("r2 sees the same schema: {:?}", r2.columns); }

Three handles, one Schema allocation. That’s how a million records carry “their” schema for free.

Completion: predict the count after a drop

This program clones, then drops one handle inside an inner scope. Two println!s are written for you; the reference count for the third is left as a gap. Fill in the number a drop produces.

rust // editable

reset ▶ run

use std::sync::Arc;

#[derive(Debug)]
struct Schema { columns: Vec<String> }

fn main() {
  let schema = Arc::new(Schema { columns: vec!["id".into()] });
  println!("start: {}", Arc::strong_count(&schema)); // 1

  {
      let _borrowed = Arc::clone(&schema);
      println!("inside scope: {}", Arc::strong_count(&schema)); // 2
  } // _borrowed drops here — the count goes back down

  // The inner handle is gone. What does strong_count report now?
  let expected = 0; // <- TODO: replace 0 with the count after the drop
  println!("after scope: {} (you predicted {})", Arc::strong_count(&schema), expected);
}

use std::sync::Arc; #[derive(Debug)] struct Schema { columns: Vec<String> } fn main() { let schema = Arc::new(Schema { columns: vec!["id".into()] }); println!("start: {}", Arc::strong_count(&schema)); // 1 { let _borrowed = Arc::clone(&schema); println!("inside scope: {}", Arc::strong_count(&schema)); // 2 } // _borrowed drops here — the count goes back down // The inner handle is gone. What does strong_count report now? let expected = 0; // <- TODO: replace 0 with the count after the drop println!("after scope: {} (you predicted {})", Arc::strong_count(&schema), expected); }

💡 Hint 1

Every Arc::clone adds one to the count; every drop subtracts one. The inner handle was created (+1) and then dropped at the end of the scope (−1). What does that leave?

Show solution

let expected = 1; // back to one owner: the original `schema`

The count is symmetric: the inner Arc::clone raised it to 2, and dropping _borrowed at the closing brace lowered it back to 1. When that last handle finally drops at the end of main, the count hits 0 and the single Schema allocation is freed, exactly once. This is the “freed when the last owner drops” rule doing its bookkeeping in front of you.

Faded: write the sharing yourself

Your turn with much less scaffolding. Build one Source on the heap behind an Arc, hand out two more owning handles so three exist at once, and print the live count (it should read 3). The skeleton gives you the struct and main; you write the body.

rust // editable

reset ▶ run

use std::sync::Arc;

#[derive(Debug)]
struct Source { path: String }

fn main() {
  // TODO:
  // 1. Put a Source on the heap behind an Arc (Arc::new).
  // 2. Make two more owning handles with Arc::clone.
  // 3. Print Arc::strong_count — it should be 3.
  todo!()
}

use std::sync::Arc; #[derive(Debug)] struct Source { path: String } fn main() { // TODO: // 1. Put a Source on the heap behind an Arc (Arc::new). // 2. Make two more owning handles with Arc::clone. // 3. Print Arc::strong_count — it should be 3. todo!() }

Show solution

fn main() {
    let source = Arc::new(Source { path: "data/input.csv".into() });

    let a = Arc::clone(&source); // handle #2 — pointer + count bump
    let b = Arc::clone(&source); // handle #3

    println!("owners: {}", Arc::strong_count(&source)); // 3
    println!("a: {:?}", a.path);
    println!("b: {:?}", b.path);
}

One Source allocation, three owners, no copy of the path string. Swap Arc for Rc here and it still compiles, because nothing crosses a thread. The moment a value like this has to move to another thread (as every Clinker record does), only the Arc version survives the compiler. That single constraint is what fixes the engine’s choice.

Why-bridge: why the engine must use Arc, not Rc

Why does the engine pay for atomic counting everywhere it shares, when Rc is cheaper? Because Clinker’s records and the data they share cross thread boundaries by design. Each source is read on its own thread, and heavy operators run on a worker pool, so a record handed from a reader thread to a worker thread carries its Arc<Schema> and Arc<str> source filename across that boundary. A value must be Send to make that move, and an Rc, with its plain non-atomic count, is !Send. It would not compile in a record.

Retrieval checkpoint

✓ Checkpoint

// quick check

What happens when you clone an Arc<Schema>?

• The whole Schema is deep-copied
• A pointer is copied and the reference count is incremented; no schema data is copied
• A new empty Schema is created

// quick check

Why does Clinker share data with Arc rather than Rc?

• Arc is always faster than Rc
• Records and shared data must cross threads (per-source threads, a worker pool), and only Arc's atomic refcount is thread-safe; an Rc is !Send and would not compile
• Rc cannot be cloned

// quick check

An engineer says: "Box, Rc, and Arc are interchangeable; they all just put a value on the heap, so pick whichever." Which correction is right?

• They are interchangeable; the engine's choice of Arc is arbitrary
• They differ by OWNERSHIP, not just heap placement: Box is single-owner, Rc is many-owner single-thread, Arc is many-owner across threads, and the engine needs cross-thread sharing, so only Arc fits
• Box is the only one that uses the heap; Rc and Arc keep their value on the stack

// quick check

Spaced recall from the collections & iterators lesson: a Record holds its cells as a Vec<Value> but its schema as an Arc<Schema>. Why the Arc on the schema specifically, and not on the Vec?

• The Vec is small and the Schema is large, so only the Schema needs the heap
• Every row needs its OWN cell values, but all rows of a file share the SAME schema: Arc lets one Schema be owned by many records cheaply, while each Vec<Value> stays per-record
• Arc is required on any field of a struct that derives Debug

You should be able to:

• You can name what Box, Rc, and Arc each provide (one heap owner; many single-thread owners; many cross-thread owners)
• You can explain why cloning an Arc<Schema> is O(1): a pointer + refcount bump, not a schema copy
• You can say why the engine uses Arc and not Rc (records cross threads; Rc is !Send)
• You wrote a program that shares one heap value across three owners and read strong_count

Verify in the checkout:

grep -n 'Arc<Schema>' crates/clinker-record/src/record/mod.rs
grep -n 'struct DocumentContext' crates/clinker-record/src/document_context.rs

Connections

You can now name the three smart pointers, explain why an Arc clone is cheap, and say why the engine is forced toward Arc. The final piece of the data layer ties ownership, borrowing, and sharing together: lifetimes, and the zero-copy reads they make safe.

Prerequisites

• Streaming records

→

Leads to

• Zero-copy reads

Go deeper on the Rust (optional, for the same concepts taught from first principles in The Rust Book):

• Want Box and the stack/heap split from first principles, on a standalone type? Read the docs →
• Want Arc, reference counts, and the Send/!Send distinction drilled with no engine framing? Read the docs →

Glossary terms used: heap, Box, Arc, reference count, shared ownership.

Mark this lesson complete