FieldStr — a string in 24 bytes

Every record field in Clinker is a FieldStr, and Value::String is the dominant payload in an ETL stream, so this one type’s width sets the per-Value cost, the very number that drives the RSS and spill thresholds from the last four lessons. This is the lowest the data layer goes. You met the 32-byte cell budget back in Data & Representation; FieldStr is how a string fits inside it: three different ownership strategies (inline, shared, unique) hand-packed into a single 24-byte type. The Rust tool that makes that packing possible is unsafe. This is the engine’s deepest use of it, and, more usefully, a model of how to use it responsibly: a tiny unsafe core, fenced in by layout assertions and documented invariants. If you want unsafe Rust from first principles, The Rustonomicon is the canonical treatment; here we use just enough to read the real type.

Prerequisites

• Bounded memory in action

→

Leads to

• Benchmark & measure memory

What you’ll be able to do

• Name FieldStr’s three arms and say what each one optimizes: inline, shared (Arc<str>), unique (Box<str>).
• Read an unsafe block guarded by a // SAFETY: comment and state the invariant the comment is asserting.
• Explain why reading a union field requires unsafe, and how the compile-time layout assertions keep that unsafe sound across future edits.
• Write a runnable unsafe call whose SAFETY comment names a true invariant, the same move FieldStr::as_str makes for its inline arm.

New terms in this lesson (each is also expanded inline at the point you first need it):

• unsafe: operations the compiler can’t verify; you promise the invariant instead.
• invariant: a fact that must always hold; the SAFETY comment writes it down.
• soundness: no reachable program state can make the unsafe misbehave.
• FieldStr: the 24-byte, three-arm string behind every record field.

Predict first

✋ Predict first

FieldStr stores three different string strategies (short bytes in place, a shared Arc<str>, and a unique Box<str>), yet size_of::<FieldStr>() is exactly 24 bytes, within Value's 32-byte cell budget. Two questions before you reveal: (1) how can three layouts share 24 bytes, and (2) which single Rust operation in the read path is the one that forces the whole type to use `unsafe`?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

(1) The arms overlap. The three strategies don’t sit side by side. They share one 24-byte slot via a union, so only one arm is live at a time. A trailing tag byte at the same offset in every arm says which. Short values fit entirely in those bytes (inline); long ones store a pointer + length and live on the heap, behind an Arc<str> when they repeat or a Box<str> when they’re unique.

(2) Reading a union field is the unsafe operation. The compiler can’t know which arm is currently valid, so every read of a union field is unsafe. That single fact is the entire reason the type needs unsafe at all.

The trap is reading “unsafe” as “unchecked, hope for the best.” That’s the wrong mental model. Here unsafe means the compiler can’t verify this, so I am asserting an invariant I’m responsible for, and, as you’ll see, every one of those invariants is either written into a // SAFETY: comment or pinned by a compile-time assertion. Hold both answers; the rest of the lesson earns them.

Three strategies in one 24-byte type

A field string is one of three things, and which one is a memory-cost decision:

• inline: short values (≤ 23 bytes: ids, codes, flags) live in the struct, zero heap allocation;
• shared: longer values that repeat or get cloned across stages are Arc<str>-backed, so a clone is an O(1) refcount bump, not a copy;
• unique: long values flagged unique by the schema are Box<str>-backed, dropping the ~16-byte Arc refcount header that’s pure waste when a value never repeats (UUIDs, addresses, notes).

All three share one 24-byte footprint via a private union, discriminated by a single trailing tag byte:

clinker-record ·field_str.rs ·FieldStr type @19acdcb4

#[repr(C)]
union Repr {
    inline: Inline,       // [u8; 23] of UTF-8 + a tag byte
    heap: HeapHeader,     // ptr + len + padding + the same tag byte
}

pub struct FieldStr {
    repr: Repr,           // inline bytes, an Arc<str> (shared), or a Box<str> (unique)
}

 FieldStr — 24 bytes, two overlapping arms (a union), ONE shared tag byte:

  inline:  [ 23 bytes of UTF-8 ........................ ][ tag = len 0..=23 ]
  heap:    [ ptr: *const u8 ][ len: usize ][ padding ...][ tag = 0xFF/0xFE  ]
                                                            ^ same offset (INLINE_CAP) in both arms

The trailing byte does double duty: for the inline arm it’s the length (always ≤ 23); for the heap arms it’s a sentinel (0xFF shared, 0xFE unique) chosen above 23 so it can never be mistaken for an inline length. That overlap is the trick that hits 24 bytes, and it’s the reason unsafe is unavoidable: reading a field of a union is unsafe, because the compiler can’t know which arm is currently valid.

clinker-record ·field_str.rs ·INLINE_CAP doc @19acdcb4

unsafe, fenced by a SAFETY contract

Here’s the read path. as_str checks the tag, then reads the matching arm, each read in an unsafe block prefaced by a // SAFETY: comment stating the invariant that makes it sound:

pub fn as_str(&self) -> &str {
    let tag = self.tag();
    if tag <= INLINE_CAP as u8 {
        // SAFETY: inline arm — `tag` is the byte length (<= INLINE_CAP), and the
        // constructor only ever wrote valid UTF-8 into `data[..len]`.
        unsafe {
            let inline = &self.repr.inline;
            std::str::from_utf8_unchecked(&inline.data[..tag as usize])
        }
    } else {
        // SAFETY: heap arm — ptr/len came from Arc::into_raw / Box::into_raw on a
        // `str`, so the bytes are live, owned by this FieldStr, and valid UTF-8.
        unsafe {
            let heap = &self.repr.heap;
            let bytes = std::slice::from_raw_parts(heap.ptr, heap.len);
            std::str::from_utf8_unchecked(bytes)
        }
    }
}

unsafe doesn’t mean “unchecked and hope.” It means the compiler can’t verify this, so I’m asserting an invariant I’m responsible for. The SAFETY comment is that assertion written down. The Drop impl is the same discipline in reverse: it reconstitutes the Arc/Box from the raw pointer to release exactly one ownership token, with a SAFETY note that each heap arm owns exactly one such token.

What makes it actually sound: layout assertions

A SAFETY comment is a promise; the layout assertions are how the promise is kept. The whole design rests on the tag byte sitting at the same offset in both union arms, so reading the tag through either arm reads the live discriminant. A compile-time block pins exactly that:

const _: () = {
    assert!(std::mem::size_of::<FieldStr>() == 24);          // the Value-budget invariant
    assert!(std::mem::offset_of!(Inline, tag) == INLINE_CAP);    // tag offset coincides...
    assert!(std::mem::offset_of!(HeapHeader, tag) == INLINE_CAP); // ...in both arms
    assert!(TAG_SHARED as usize > INLINE_CAP);               // sentinels can't be inline lengths
    assert!(TAG_UNIQUE as usize > INLINE_CAP);
};

If anyone changes a field and breaks the layout, the build fails: the unsafe code can never run against a layout it wasn’t written for. And because the auto-derived Send/Sync can’t see through the raw pointer in the union, those are hand-written unsafe impls with their own SAFETY note (every arm’s backing is Send + Sync; the raw pointer is only ever an owned Arc/Box in disguise). A runtime test pins the size too:

clinker-record ·field_str.rs ·size_is_24_bytes test @19acdcb4

This is the responsible-unsafe pattern, and it’s worth internalising: keep the unsafe core small, pin every layout invariant it relies on with a static assertion, document each block’s SAFETY contract, and back it with a drop/round-trip test. That discipline is what gives the module its soundness. Unsafe earns its keep here (one allocation saved per short field, across billions of fields) precisely because it’s this tightly fenced.

Worked → completion → faded

You can’t write a union in a quick playground, but you can practise the core move at the heart of every arm of as_str: skip a check you can prove is unnecessary, and write the proof down as a SAFETY comment. We build that up one rung at a time.

Worked: the unchecked move, fully annotated

Here is the inline arm’s move on its own. The bytes really are valid UTF-8, so the validation scan from_utf8 would run is provably redundant, and from_utf8_unchecked skips it. Run it and confirm the unchecked and checked results agree.

rust // editable

reset ▶ run

fn main() {
  // These bytes are an ASCII literal, so they ARE valid UTF-8.
  let bytes: &[u8] = b"active";

  // SAFETY: `bytes` is a string literal — guaranteed valid UTF-8 — so skipping
  // the validation scan is sound. This is exactly what FieldStr::as_str does for
  // its inline arm, where the constructor guarantees only valid UTF-8 was written.
  let s: &str = unsafe { std::str::from_utf8_unchecked(bytes) };
  println!("unchecked: {s}");

  // The safe version does the same work but pays a validation pass:
  let checked: &str = std::str::from_utf8(bytes).unwrap();
  println!("checked:   {checked}");
  println!("agree: {}", s == checked);
}

fn main() { // These bytes are an ASCII literal, so they ARE valid UTF-8. let bytes: &[u8] = b"active"; // SAFETY: `bytes` is a string literal — guaranteed valid UTF-8 — so skipping // the validation scan is sound. This is exactly what FieldStr::as_str does for // its inline arm, where the constructor guarantees only valid UTF-8 was written. let s: &str = unsafe { std::str::from_utf8_unchecked(bytes) }; println!("unchecked: {s}"); // The safe version does the same work but pays a validation pass: let checked: &str = std::str::from_utf8(bytes).unwrap(); println!("checked: {checked}"); println!("agree: {}", s == checked); }

The unsafe block is sound because the SAFETY comment’s claim is true: the bytes really are valid UTF-8. Feed from_utf8_unchecked genuinely invalid bytes and you’d have undefined behaviour with no error; the entire responsibility moves from the compiler to you, which is why the invariant must be real and written down.

Completion: write the SAFETY invariant

Below, a constructor builds the bytes and the unsafe call is already there, but the // SAFETY: line is a TODO. A SAFETY comment must name the specific invariant that makes the unchecked call sound: here, why are bytes guaranteed to be valid UTF-8? Fill it in.

rust // editable

reset ▶ run

fn main() {
  // We build the bytes by encoding a real `&str`, so by construction they are
  // a valid UTF-8 sequence — nothing else was ever written into the buffer.
  let mut buf: Vec<u8> = Vec::new();
  buf.extend_from_slice("hello".as_bytes());
  buf.extend_from_slice(" world".as_bytes());

  // SAFETY: TODO — name the invariant. Why are the bytes in `buf` valid UTF-8?
  let s: &str = unsafe { std::str::from_utf8_unchecked(&buf) };

  println!("{s}");
}

fn main() { // We build the bytes by encoding a real `&str`, so by construction they are // a valid UTF-8 sequence — nothing else was ever written into the buffer. let mut buf: Vec<u8> = Vec::new(); buf.extend_from_slice("hello".as_bytes()); buf.extend_from_slice(" world".as_bytes()); // SAFETY: TODO — name the invariant. Why are the bytes in `buf` valid UTF-8? let s: &str = unsafe { std::str::from_utf8_unchecked(&buf) }; println!("{s}"); }

💡 Hint 1

A SAFETY comment points at the source of the guarantee, not at “trust me”. Here the only thing ever written into buf came from str::as_bytes(). What does that guarantee about the bytes?

Show solution

// SAFETY: every byte in `buf` was copied from `&str::as_bytes()`, which always
// yields valid UTF-8, and nothing else was written — so the whole buffer is a
// valid UTF-8 sequence and skipping the validation scan is sound.

The shape matches FieldStr::as_str exactly: the safety of the unchecked read rests entirely on a fact the constructor established (only valid UTF-8 was ever stored). The comment’s job is to name that fact so a future reader (or a future you, editing the constructor) knows precisely what must stay true.

Faded: write the guarded function

Your turn with much less scaffolding. Write first_word, which returns the first space-delimited word of a &str as a &str, using from_utf8_unchecked on the relevant byte slice. The exercise is about one thing: you supply the SAFETY comment naming a true invariant, the way every arm of as_str does. (A safe from_utf8(...).unwrap() would also work; here you deliberately practise the unsafe move with its proof, because that’s the discipline the engine relies on.)

rust // editable

reset ▶ run

// TODO: return the first word of `line` (everything up to the first space, or
// the whole string if there's no space). Slice the bytes, then build the &str
// with std::str::from_utf8_unchecked — and write a SAFETY comment that names
// WHY those bytes are valid UTF-8.
fn first_word(line: &str) -> &str {
  todo!()
}

fn main() {
  println!("{:?}", first_word("active record"));
  println!("{:?}", first_word("solo"));
}

// TODO: return the first word of `line` (everything up to the first space, or // the whole string if there's no space). Slice the bytes, then build the &str // with std::str::from_utf8_unchecked — and write a SAFETY comment that names // WHY those bytes are valid UTF-8. fn first_word(line: &str) -> &str { todo!() } fn main() { println!("{:?}", first_word("active record")); println!("{:?}", first_word("solo")); }

Show solution

fn first_word(line: &str) -> &str {
    let bytes = line.as_bytes();
    let end = bytes.iter().position(|&b| b == b' ').unwrap_or(bytes.len());

    // SAFETY: `line` is already a &str (valid UTF-8), and a space (0x20) is an
    // ASCII byte — never the middle of a multi-byte char — so `end` lands on a
    // char boundary and `bytes[..end]` is itself a valid UTF-8 sequence.
    unsafe { std::str::from_utf8_unchecked(&bytes[..end]) }
}

The invariant here is subtler than “it came from a &str”: it’s that the split point lands on a char boundary. Because a space is a single ASCII byte, it can never appear inside a multi-byte UTF-8 sequence, so slicing at it is safe. Naming that, rather than settling for “trust me”, is what makes the SAFETY comment do real work. This is the same care FieldStr takes: the inline arm only slices at tag because the constructor guarantees data[..tag] is a whole, valid UTF-8 string.

That deal unsafe offers (you keep the invariant true, the compiler steps aside) is the one FieldStr takes only where the payoff is large and the invariant is pinned by an assertion.

Comfortable with unsafe, unions, and the SAFETY discipline? Skip ahead →

// quick check

What makes FieldStr's unsafe union reads sound rather than reckless?

• unsafe blocks are checked more strictly by the compiler
• A small unsafe core whose every invariant is pinned: layout assertions force the tag to a fixed offset (so reading it is always valid), each block has a SAFETY comment stating its invariant, and constructors guarantee only valid UTF-8 / proper ownership tokens are stored
• The union can only ever hold one arm so no checks are needed

Retrieval checkpoint

✓ Checkpoint: unsafe & FieldStr

💡 Hint 1

Count the // SAFETY: comments; each one fences one unsafe block. Then read the const _: () block: which invariant does it pin, and why does the union read depend on it?

What the tour establishes

FieldStr is a 24-byte union of an inline arm and a heap arm, discriminated by a trailing tag byte. Reading a union field is unsafe; each read is fenced by a SAFETY comment stating its invariant (valid UTF-8, or a live owned pointer). The const _: () assertions pin the size and the tag offset the unsafe depends on, so a layout-breaking edit fails the build; size_is_24_bytes pins it at run time too. Small unsafe core, every invariant asserted: that’s why it’s sound.

// quick check

The tag byte's sentinels (0xFF, 0xFE) are chosen above INLINE_CAP (23). Why does that matter?

• To make the struct exactly 24 bytes
• So a heap-arm sentinel can never be confused with a valid inline length (0..=23): one byte unambiguously says 'inline of length N' or 'shared/unique', which is what lets the union be discriminated safely
• Because Arc requires those specific values

// quick check

You write `unsafe { std::str::from_utf8_unchecked(bytes) }` and the program prints fine in your tests. Which statement is the MISCONCEPTION to correct?

• If it printed correctly on my inputs, the unsafe block is proven sound
• Soundness means NO possible input can trigger undefined behaviour; passing on the inputs I tried proves nothing, and the invariant (bytes are valid UTF-8) must hold for every reachable call
• The SAFETY comment must name the invariant that guarantees the bytes are valid UTF-8

// quick check

Spaced recall from the sharing-and-boxing lesson: FieldStr's 'unique' arm uses Box<str> while its 'shared' arm uses Arc<str>. Why is Box the right choice for a value that never repeats?

• Box stores the bytes inline in the 24 bytes, avoiding the heap entirely
• A Box is a single owning pointer with no reference count, so it drops the ~16-byte Arc refcount header that's pure waste when a value is never shared or cloned
• Box is faster to read than Arc because it skips UTF-8 validation

You should be able to:

• You can name FieldStr's three arms and what each optimizes (inline, Arc-shared, Box-unique)
• You can explain why reading a union field needs unsafe and what a SAFETY comment asserts
• You can explain how the layout assertions keep the unsafe sound across future edits
• You can write an unsafe from_utf8_unchecked call whose SAFETY comment names a true invariant

Verify in the checkout:

grep -n 'union Repr\|pub struct FieldStr' crates/clinker-record/src/field_str.rs
grep -n 'SAFETY' crates/clinker-record/src/field_str.rs | head
grep -n 'const _: ()' crates/clinker-record/src/field_str.rs
cargo test -p clinker-record --lib -- size_is_24_bytes

Connections

You’ve reached the floor of the data layer. The final Execution & Memory lesson, Benchmark & measure memory, turns the lens around: how do you measure all this, the per-value cost model, the benchmark suite, and a custom allocator that counts every byte?

Go deeper on the Rust (optional, one-directional, for the building blocks FieldStr’s arms are made of, taught from first principles in The Rust Book):

• Want Box (the single heap owner behind the unique arm) from first principles? Read the docs →
• Want Arc and shared ownership (the shared arm) drilled standalone? Read the docs →

Mark this lesson complete