Testing strategy

Welcome to Extending & Contributing. The earlier modules, from Orientation onward, taught you to read the engine; this one turns you toward changing it. And the first thing a contributor needs is a way to know they didn’t break anything, well before any clever feature. This lesson is the engine’s answer to “how do I prove a change is correct?”, and it does it with four different kinds of test, each aimed at a failure the others would miss.

Prerequisites

• Benchmark & measure memory

→

Leads to

• Add a CXL builtin

What you’ll be able to do

• Distinguish an inline unit test from a tests/ integration test, and name what each one can reach.
• Read a boundary integration test and explain why it survives an internal refactor that a unit test would not.
• Trace the compare_or_write golden-baseline control flow and say what UPDATE_BASELINES=1 does on the first run versus every run after.
• Write a property test that checks a fast function against a slow, self-evidently-correct oracle, and say when that machinery earns its keep.

New terms in this lesson (each is also expanded inline at the point you first need it):

• unit test: an inline test that can reach a file’s private internals.
• integration test: a tests/ test that can only touch the public API.
• golden / snapshot test: assert against a committed reference output instead of a hand-written literal.
• property test: generate hundreds of random inputs and check an invariant on each.
• fixture: a committed input/expected file the test loads.
• oracle: a slow, trusted twin you check a fast implementation against.

Predict first

✋ Predict first

You refactor the internals of the aggregation operator (rename private helpers, change a data structure) but intend ZERO change to its output. You have two tests: (a) an inline unit test that calls a private helper you just renamed, and (b) a tests/ integration test that feeds YAML + CSV to the public executor and asserts on the output bytes. After the refactor, which test still even COMPILES, and which one most directly proves you didn't change behavior? Commit to an answer for each before revealing.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

The integration test (b) wins on both counts, and seeing why is the spine of this lesson.

• The unit test (a) names a private helper you just renamed, so it may not even compile until you edit the test too. Worse, a unit test on internals is pinned to the implementation: it changes whenever the implementation changes, which is exactly the moment you most want a test that didn’t.
• The integration test (b) can only touch the public API. It feeds eng,100 plus eng,200 and asserts the output is eng,300,2. Rewrite the operator however you like; as long as that still sums correctly, the test stays green, and it goes red the instant behavior actually drifts.

The wrong mental model is “more tests of the internals = more safety.” A test welded to internals is a test that breaks on harmless refactors and stays silent on the one that matters. The most durable test is the one pinned to the contract, not the code. Hold that idea; every tier below is a variation on it.

Two tiers: unit tests next to the code, integration tests at the seam

Clinker’s tests live in two clearly separated places, and the split is the first thing to internalize:

• An inline unit test is a #[cfg(test)] mod tests block at the bottom of a source file, testing that file’s internals. It can reach private functions.
• An integration test is a file under crates/<crate>/tests/, compiled as a separate crate that can only call the crate’s public API.

A small inline unit test, right beside the coercion logic it covers:

clinker-record ·coercion.rs ·test_coerce_string_to_int_valid test @19acdcb4

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_coerce_string_to_int_valid() {
        // exercises coerce_to_int directly — a private-ish unit of behavior
    }
}

The naming is deliberate and worth copying: tests read as module::tests::scenario::behavior, so a failure name tells you what broke before you open the file. The testing-commands doc records the convention and the canonical run commands:

clinker ·50_TESTING_AND_COMMANDS.md doc @19acdcb4

# Fast signal after an edit — does it still compile?
cargo check --workspace --locked --offline

# Run ONE test exactly (the period-separated path is the full test name):
cargo test -p clinker-exec --lib --offline \
  executor::tests::spill_dir_unavailable_midrun::unarmed_seam... -- --exact

# The whole suite. The ulimit prefix is load-bearing: the default 1024-fd
# limit makes clinker-exec's spill tests fail with "Too many open files".
ulimit -n 4096 && cargo test --workspace --locked --offline

Evidence note

The -- --exact targeted-run form and the ulimit -n 4096 prefix are quoted from docs/ai/50_TESTING_AND_COMMANDS.md (Documented). The phrase “test to the boundary” below is not a documented slogan; it’s the name we’re giving a pattern that’s clearly visible in the integration-test structure (Observed).

Testing to the boundary

An integration test under tests/ can’t see internals, so it’s forced to drive the engine the way a user does: feed YAML and input bytes to the public executor, then assert on the output bytes and the run report. That’s “testing to the boundary,” and it’s the engine’s most valuable kind of test, because it survives any internal refactor that keeps the public behavior the same.

clinker-exec ·aggregate_integration.rs ·test_e2e_group_by_sum_count test @19acdcb4

// End-to-end: CSV in → aggregate(group_by:[dept], sum + count) → CSV out.
let csv = "dept,salary\neng,100\neng,200\nsales,50\n";
let (report, output) = run_single(yaml, csv);

assert_eq!(report.dlq_entries.len(), 0);
assert_eq!(report.counters.ok_count, 2, "two output groups");
assert_eq!(
    sorted_body_lines(&output),
    vec!["eng,300,2".to_string(), "sales,50,1".to_string()],
);

Nothing here names a private type. If someone rewrites the aggregation operator’s internals tomorrow, this test still passes as long as eng,100 plus eng,200 still sums to eng,300,2. That’s the whole point: the assertion is pinned to the contract, not the implementation, exactly the prediction you made up top.

Snapshot tests: assert on a big blob without hand-writing it

Some outputs are too large to hand-author an assert_eq! for, like the full text of an execution plan from --explain. A snapshot test solves that: clinker uses the insta crate so you write the test, run it once, and insta records the output as a committed .snap file. Later runs compare against that file; an intentional change is reviewed and re-accepted.

clinker-exec ·cull_explain_snapshot.rs ·explain_renders_cull_two_output_ports test @19acdcb4

#[test]
fn explain_renders_cull_two_output_ports() {
    let text = render_explain(yaml);
    // a few structural asserts first (these document intent) ...
    assert!(text.contains("FORK [cull] 'drop_bad'"));
    // ... then snapshot the whole rendered plan under a stable name:
    insta::assert_snapshot!("explain_cull_two_output_ports", text);
}

The committed snapshot it locks against starts with an insta header and then the captured value:

---
source: crates/clinker-exec/tests/cull_explain_snapshot.rs
expression: text
---
=== Execution Plan ===

Mode: Streaming
...

When you intentionally change --explain output, the snapshot test fails, you eyeball the diff, and accept it with cargo insta review (or INSTA_UPDATE=always). The discipline: a snapshot diff in a PR is a visible, reviewable record of an output-format change. It can’t slip through silently.

Tooling vs. documented convention

cargo insta review / INSTA_UPDATE are standard insta tooling, not commands written down in clinker’s docs. Clinker’s own update knob is UPDATE_BASELINES=1, for the golden CSVs in the next section, a different mechanism for a different fixture kind.

Golden-baseline regression seeds

The strongest refactor net in the codebase is a corpus of golden baselines: real pipelines whose exact output bytes are committed as a fixture under tests/fixtures/baselines/ (e.g. csv_transform_sink.expected.csv). A driver runs each pipeline and compares the fresh output to the committed file, byte for byte:

clinker-exec ·pre_lift_baselines.rs ·compare_or_write fn @19acdcb4

fn compare_or_write(baseline_name: &str, actual: &str) {
    let p = baseline_root().join(baseline_name);
    if update_mode() || !p.exists() {
        // First run (or UPDATE_BASELINES=1): capture the golden.
        std::fs::write(&p, actual.as_bytes()).unwrap();
        return;
    }
    let expected = std::fs::read_to_string(&p).unwrap();
    assert_eq!(actual, expected, "byte-mismatch against baseline {}", p.display());
}

Read the control flow carefully; it’s the whole regression-seed pattern in ten lines. The first time a fixture runs (or whenever you deliberately set UPDATE_BASELINES=1), the current output is written as the new golden. Every run after that compares. So the seed is captured once, then frozen; any future change that alters a single byte of any baseline pipeline’s output trips a named failure. (Note: the corpus is keyed by fixture name, not by issue number, and clinker doesn’t tag regression tests with bug IDs.)

One property test: fast algorithm vs. slow oracle

Most tests check fixed examples. A property test instead generates hundreds of random inputs and checks an invariant on every one. Clinker uses proptest for exactly one high-value case: its fast band-join (iejoin_numeric) must agree with a dead-simple, self-evidently-correct nested-loop join on every random input.

clinker-exec ·iejoin.rs ·proptest_iejoin_matches_nested_loop test @19acdcb4

proptest! {
    #![proptest_config(ProptestConfig::with_cases(256))]
    #[test]
    fn proptest_iejoin_matches_nested_loop((left, right, op1, op2) in arb_inputs()) {
        let actual: HashSet<(usize, usize)> =
            iejoin_numeric(&left, &right, op1.to_range(), op2.to_range())
                .into_iter().collect();
        let expected = nested_loop(&left, &right, op1, op2);   // the slow oracle
        prop_assert_eq!(actual, expected);
    }
}

This is the oracle pattern: you have a fast implementation you’re unsure about and a slow implementation you trust, and you assert they always agree. It’s worth the machinery precisely because the fast path (coarse-filter striding, permutation indexing) is the kind of code that’s prone to subtle bugs. For straightforward behavior, a handful of example tests is cheaper and clearer, so don’t reach for proptest by default. (The repo also has a combine_iejoin_prop.rs scaffold; the live property test is the inline one shown here.)

Worked → completion → faded

You’ve read all four tiers. Now you write the highest-value one, the oracle property test, on a self-contained example, scaffolded down to your own. Each <Playground> below is runnable Rust on its own; none needs clinker.

Worked: an oracle test, fully written

Here is the oracle pattern in miniature, end to end. The “fast” function under test is a hand-rolled binary search; the “slow oracle” is a linear scan no one doubts. The property: both must return the same answer for every input. This version uses no test framework: it just loops over random-ish inputs and asserts, so you can run it and watch it pass.

rust // editable

reset ▶ run

// FAST path under test: binary search for the first index whose value >= target.
fn lower_bound(sorted: &[i64], target: i64) -> usize {
  let (mut lo, mut hi) = (0, sorted.len());
  while lo < hi {
      let mid = lo + (hi - lo) / 2;
      if sorted[mid] < target { lo = mid + 1; } else { hi = mid; }
  }
  lo
}

// SLOW ORACLE: self-evidently correct — just count the elements strictly below target.
fn lower_bound_oracle(sorted: &[i64], target: i64) -> usize {
  sorted.iter().filter(|&&x| x < target).count()
}

fn main() {
  // A tiny deterministic generator stands in for proptest's random cases.
  let mut seed: u64 = 0x9E3779B97F4A7C15;
  let mut next = || { seed ^= seed << 13; seed ^= seed >> 7; seed ^= seed << 17; seed };

  for _ in 0..2000 {
      let len = (next() % 8) as usize;
      let mut v: Vec<i64> = (0..len).map(|_| (next() % 20) as i64).collect();
      v.sort();                                   // lower_bound requires sorted input
      let target = (next() % 22) as i64 - 1;      // -1..=20, includes out-of-range

      let fast = lower_bound(&v, target);
      let slow = lower_bound_oracle(&v, target);
      // The invariant: fast and slow ALWAYS agree.
      assert_eq!(fast, slow, "mismatch on {v:?} target={target}");
  }
  println!("all 2000 random cases agree: fast == oracle");
}

// FAST path under test: binary search for the first index whose value >= target. fn lower_bound(sorted: &[i64], target: i64) -> usize { let (mut lo, mut hi) = (0, sorted.len()); while lo < hi { let mid = lo + (hi - lo) / 2; if sorted[mid] < target { lo = mid + 1; } else { hi = mid; } } lo } // SLOW ORACLE: self-evidently correct — just count the elements strictly below target. fn lower_bound_oracle(sorted: &[i64], target: i64) -> usize { sorted.iter().filter(|&&x| x < target).count() } fn main() { // A tiny deterministic generator stands in for proptest's random cases. let mut seed: u64 = 0x9E3779B97F4A7C15; let mut next = || { seed ^= seed << 13; seed ^= seed >> 7; seed ^= seed << 17; seed }; for _ in 0..2000 { let len = (next() % 8) as usize; let mut v: Vec<i64> = (0..len).map(|_| (next() % 20) as i64).collect(); v.sort(); // lower_bound requires sorted input let target = (next() % 22) as i64 - 1; // -1..=20, includes out-of-range let fast = lower_bound(&v, target); let slow = lower_bound_oracle(&v, target); // The invariant: fast and slow ALWAYS agree. assert_eq!(fast, slow, "mismatch on {v:?} target={target}"); } println!("all 2000 random cases agree: fast == oracle"); }

The shape is the whole lesson: an optimization you’re unsure of, a dead-simple version you trust, and an assertion that they never disagree across many inputs.

Completion: fill in the invariant

Below, the generator and both implementations are written for you. This time the fast path is a dedup_count (count distinct values) and the oracle uses a HashSet. One line is missing: the assertion that states the property. Fill it in so a mismatch fails loudly with the offending input.

rust // editable

reset ▶ run

use std::collections::HashSet;

// FAST path under test: count distinct values assuming the slice is already sorted.
fn distinct_sorted(sorted: &[i64]) -> usize {
  if sorted.is_empty() { return 0; }
  let mut count = 1;
  for w in sorted.windows(2) {
      if w[0] != w[1] { count += 1; }
  }
  count
}

// SLOW ORACLE: drop everything into a set and ask its size. Order-independent, trusted.
fn distinct_oracle(values: &[i64]) -> usize {
  values.iter().copied().collect::<HashSet<i64>>().len()
}

fn main() {
  let mut seed: u64 = 0xD1B54A32D192ED03;
  let mut next = || { seed ^= seed << 13; seed ^= seed >> 7; seed ^= seed << 17; seed };

  for _ in 0..2000 {
      let len = (next() % 10) as usize;
      let mut v: Vec<i64> = (0..len).map(|_| (next() % 6) as i64).collect();
      v.sort();   // distinct_sorted needs sorted input; the oracle does not care

      let fast = distinct_sorted(&v);
      let slow = distinct_oracle(&v);
      // TODO: assert the property — fast and slow must agree, and on failure
      //       the message must show the input that broke it.
  }
  println!("all cases agree: fast == oracle");
}

use std::collections::HashSet; // FAST path under test: count distinct values assuming the slice is already sorted. fn distinct_sorted(sorted: &[i64]) -> usize { if sorted.is_empty() { return 0; } let mut count = 1; for w in sorted.windows(2) { if w[0] != w[1] { count += 1; } } count } // SLOW ORACLE: drop everything into a set and ask its size. Order-independent, trusted. fn distinct_oracle(values: &[i64]) -> usize { values.iter().copied().collect::<HashSet<i64>>().len() } fn main() { let mut seed: u64 = 0xD1B54A32D192ED03; let mut next = || { seed ^= seed << 13; seed ^= seed >> 7; seed ^= seed << 17; seed }; for _ in 0..2000 { let len = (next() % 10) as usize; let mut v: Vec<i64> = (0..len).map(|_| (next() % 6) as i64).collect(); v.sort(); // distinct_sorted needs sorted input; the oracle does not care let fast = distinct_sorted(&v); let slow = distinct_oracle(&v); // TODO: assert the property — fast and slow must agree, and on failure // the message must show the input that broke it. } println!("all cases agree: fast == oracle"); }

💡 Hint 1

The oracle test’s entire job is one comparison. You want assert_eq! of the two counts, with a format-string message that interpolates the failing vector so a red test tells you which input broke.

Show solution

assert_eq!(fast, slow, "distinct mismatch on {v:?}");

That single line is the property. Because the oracle is order-independent and the fast path assumes sorted input, this test would instantly catch a bug where distinct_sorted forgot that windows need sorted data: the counts would diverge and the message would print the exact vector.

Faded: write the whole property test

Your turn with much less scaffolding. The fast function merge_count is meant to count how many elements two sorted slices have in common (a set intersection size), walking both with two pointers. Write the oracle and the property loop yourself. The oracle should be the self-evidently-correct version (two HashSets and an intersection); the loop should generate random sorted inputs and assert the two agree.

rust // editable

reset ▶ run

use std::collections::HashSet;

// FAST path under test: two-pointer intersection size over two SORTED slices.
fn merge_count(a: &[i64], b: &[i64]) -> usize {
  let (mut i, mut j, mut hits) = (0, 0, 0);
  while i < a.len() && j < b.len() {
      if a[i] < b[j] { i += 1; }
      else if a[i] > b[j] { j += 1; }
      else { hits += 1; i += 1; j += 1; }
  }
  hits
}

// TODO 1: write the slow oracle — intersect two HashSets and return the size.
//         Signature: fn merge_oracle(a: &[i64], b: &[i64]) -> usize

fn main() {
  let mut seed: u64 = 0x2545F4914F6CDD1D;
  let mut next = || { seed ^= seed << 13; seed ^= seed >> 7; seed ^= seed << 17; seed };

  // TODO 2: loop ~2000 times; build two random SORTED Vec<i64>; assert
  //         merge_count == merge_oracle, with the inputs in the failure message.

  println!("all cases agree: fast == oracle");
}

use std::collections::HashSet; // FAST path under test: two-pointer intersection size over two SORTED slices. fn merge_count(a: &[i64], b: &[i64]) -> usize { let (mut i, mut j, mut hits) = (0, 0, 0); while i < a.len() && j < b.len() { if a[i] < b[j] { i += 1; } else if a[i] > b[j] { j += 1; } else { hits += 1; i += 1; j += 1; } } hits } // TODO 1: write the slow oracle — intersect two HashSets and return the size. // Signature: fn merge_oracle(a: &[i64], b: &[i64]) -> usize fn main() { let mut seed: u64 = 0x2545F4914F6CDD1D; let mut next = || { seed ^= seed << 13; seed ^= seed >> 7; seed ^= seed << 17; seed }; // TODO 2: loop ~2000 times; build two random SORTED Vec<i64>; assert // merge_count == merge_oracle, with the inputs in the failure message. println!("all cases agree: fast == oracle"); }

Show solution

use std::collections::HashSet;

fn merge_count(a: &[i64], b: &[i64]) -> usize {
    let (mut i, mut j, mut hits) = (0, 0, 0);
    while i < a.len() && j < b.len() {
        if a[i] < b[j] { i += 1; }
        else if a[i] > b[j] { j += 1; }
        else { hits += 1; i += 1; j += 1; }
    }
    hits
}

fn merge_oracle(a: &[i64], b: &[i64]) -> usize {
    let sa: HashSet<i64> = a.iter().copied().collect();
    let sb: HashSet<i64> = b.iter().copied().collect();
    sa.intersection(&sb).count()
}

fn main() {
    let mut seed: u64 = 0x2545F4914F6CDD1D;
    let mut next = || { seed ^= seed << 13; seed ^= seed >> 7; seed ^= seed << 17; seed };

    for _ in 0..2000 {
        let make = |n: &mut dyn FnMut() -> u64| {
            let len = (n() % 8) as usize;
            let mut v: Vec<i64> = (0..len).map(|_| (n() % 12) as i64).collect();
            v.sort();
            v
        };
        let a = make(&mut next);
        let b = make(&mut next);
        assert_eq!(merge_count(&a, &b), merge_oracle(&a, &b),
                   "intersection mismatch on a={a:?} b={b:?}");
    }
    println!("all cases agree: fast == oracle");
}

Note one subtlety the oracle exposes for free: if either input slice has duplicates, merge_count’s two-pointer walk can count a shared value more than once, while the HashSet oracle counts it once. The property test would surface that divergence immediately, which is exactly why you reach for the oracle pattern on tricky algorithms and not on straight-line code. In real clinker this is the same machinery proptest runs for you, with shrinking to a minimal failing case on top.

Why-bridge: four tiers, four failure modes

Why does the engine carry four kinds of test instead of one? Because each catches a class of bug the others structurally cannot:

• A unit test catches a logic error in one private function: fast, precise, but blind to how the pieces compose.
• A boundary integration test catches a behavior change at the public contract, and it survives internal refactors, though it can’t economically pin down a 5,000-line --explain blob.
• A snapshot / golden test catches exactly that: any byte-level drift in a large output, with the diff itself as the review artifact. What it won’t tell you is which input triggered the bug, only that the output changed.
• A property test catches the input you’d never think to hand-write, by generating hundreds and checking an invariant, at the cost of machinery you only want where a fast path could be subtly wrong.

Want the engine's error-type design that the boundary test's run-report asserts against? Skip ahead →

Retrieval checkpoint

✓ Checkpoint: testing strategy

💡 Hint 1

Run the aggregate integration test first; it’s the boundary test from the tour. Then list the baselines directory: each .expected.csv is one frozen golden output. Open compare_or_write and trace the update_mode() || !p.exists() branch.

💡 Hint 2

To see the regression net work: pick a baseline pipeline, change its expected .csv by one byte, and re-run: the byte-mismatch panic names the fixture. Restore it with git restore, or regenerate with UPDATE_BASELINES=1.

What the tour establishes

Tests live in two tiers: inline #[cfg(test)] mod tests (private internals) and crates/*/tests/ integration files (public API only). The most durable tests drive the public executor and assert on output bytes: testing to the boundary. Large outputs use insta snapshots (reviewable diffs); whole-pipeline outputs use committed golden .expected.csv baselines compared by compare_or_write (capture-once via UPDATE_BASELINES=1, compare-forever). One proptest checks the fast band-join against a nested-loop oracle, the right tool when a fast algorithm needs a trusted slow twin.

// quick check

You refactor the internals of the aggregation operator but intend zero change to its output. Which test most directly protects you, and why?

• A new inline unit test on a private helper you just renamed
• The end-to-end integration test that feeds YAML + CSV and asserts on output bytes; it's pinned to the public contract, so it stays valid across any internal rewrite that preserves behavior
• The proptest, because property tests are always strongest

// quick check

What does setting UPDATE_BASELINES=1 do for a golden-baseline fixture that already has a committed .expected.csv?

• Nothing; baselines are immutable once committed
• It overwrites the committed baseline with the current run's output, so you must review that diff like any other intentional change
• It deletes the baseline and skips the test

// quick check

A teammate says: 'Every algorithm should have a proptest; property tests are strictly stronger than example tests, so example tests are wasted effort.' Which correction is right?

• Correct: once you have a proptest, delete the example tests
• Misconception: a proptest needs a trusted oracle or invariant and adds real machinery (generators, shrinking); for straight-line behavior a few example tests are cheaper, clearer, and just as safe, and clinker uses proptest for exactly one high-value case
• Misconception: property tests can't find bugs that example tests miss

// quick check

Spaced recall from the error-handling lesson: the boundary test asserts report.dlq_entries.len() == 0. Under ErrorStrategy::Continue, what would a non-zero dlq_entries count after a run mean?

• The whole run aborted on the first bad record
• Some records were recoverable per-record failures that got routed to the dead-letter queue while the run continued
• An Internal invariant error fired, which is always dead-lettered

You should be able to:

• You can state the difference between an inline #[cfg(test)] unit test and a tests/ integration test (what each can reach)
• You can explain why a boundary integration test survives an internal refactor that breaks an internals unit test
• You can explain what a golden baseline is and what UPDATE_BASELINES=1 does on first run vs later runs
• You can write an oracle property test and say when it earns its keep versus a few example tests

Verify in the checkout:

ulimit -n 4096 && cargo test -p clinker-exec --offline aggregate_integration
grep -rn 'insta::assert_snapshot' crates/clinker-exec/tests/ | head
ls crates/clinker-exec/tests/fixtures/baselines/
grep -n 'proptest_iejoin_matches_nested_loop' crates/clinker-exec/src/pipeline/iejoin.rs

Connections

You can now name the four test tiers, say which failure each catches, trace the golden-baseline control flow, and write an oracle property test of your own. Next: make your first real change, adding a builtin to CXL, the engine’s expression language.

Prerequisites

• Benchmark & measure memory

→

Leads to

• Add a CXL builtin

Go deeper in the engine (optional, one-directional, the engine’s error-type design the boundary test’s run report asserts against):

• Want to build the engine's error type by hand (Display, source, and From)? Skip ahead →

Glossary terms used: unit test, integration test, golden / snapshot test, property test, fixture, oracle.

Mark this lesson complete