Plan a change without breaking a boundary

You can now extend all three of the engine’s seams. The last skill is the one that separates a contributor from a careful contributor: knowing before you write code which architectural boundary your change touches, and recognizing the decisions that aren’t yours to make alone. This is the capstone lesson, and the bridge to the capstones themselves.

Prerequisites

• Review & the gauntlet

→

Leads to

• Engine Track — overview

What you’ll be able to do

• Name three of the engine’s architectural invariants and say where each is written down.
• Read a crate’s Cargo.toml and decide whether a proposed dependency respects the one-way layering.
• Trace the blast radius of a proposed change (which crates and boundaries it touches) before writing any code.
• Distinguish a change you can land directly from one that must go through a Decision Gate, by checking it against the trigger list.

New terms in this lesson (each is also expanded inline at the point you first need it):

• invariant: a load-bearing rule the rest of the code assumes holds.
• seam: a deliberate, contained extension point you can add to safely.
• blast radius: everything a change actually touches, traced before you start.
• change scoping: mapping the blast radius against the invariants and the gate triggers, first.

Predict first

✋ Predict first

You want to make the aggregation operator faster by having clinker-plan precompute a lookup table that the operator reads at run time. The clean-looking way to share the code is to add clinker-exec to clinker-plan's [dependencies]. Before you read on: is this (a) a fine local optimization you can just land, (b) a change that needs restructuring but no escalation, or (c) a change that must go through a Decision Gate? Write your pick and one sentence of reasoning.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

The answer is (c): it must go through a Decision Gate, and it never even gets to compile the way you’d hope.

The crate map is one-directional: clinker-plan sits below clinker-exec and must not depend on it. Adding clinker-exec to clinker-plan’s manifest creates a dependency cycle, which Cargo refuses to build. So the “clean” version is unrepresentable, the same shape as the plan/runtime boundary, where you couldn’t hand the executor a raw config.

But notice the deeper move: even wanting this is the signal. Crossing a crate boundary is an explicit gate trigger (cross-crate or cross-service architecture boundaries). The right action isn’t “find a workaround that compiles.” Copying the operator body into clinker-plan would compile and still be wrong, because it duplicates the same coupling by hand. The right action is to stop and scope it: the shared type belongs in a lower layer both crates can depend on, and that restructuring is an architecture decision you escalate rather than encode.

The tempting wrong model is (a): “it’s just a performance tweak, the blast radius is one operator.” The blast radius is never one operator the moment a change reaches across a crate boundary, and the whole point of this lesson is tracing that before you write the code.

The invariants are written down

Clinker keeps its architectural rules explicit, in docs/ai. They are the load-bearing properties everything else assumes, not style preferences. An invariant quietly broken isn’t a bug in the usual sense; it’s a boundary break the type system might not catch. A few of the most concrete (quoted):

clinker ·10_ARCHITECTURE.md ·Pipelines compile before they execute. doc @19acdcb4

Pipelines compile before they execute. Planning/config validation belongs in clinker-plan; runtime operator execution belongs in clinker-exec.

That single rule is the one you’ve now seen from four angles: the typed CompiledPlan handle (the plan/runtime boundary), the lowering step, and the plan-vs-exec crate split. Other invariants in the same register:

• The runtime is finite and synchronous: “Do not introduce unbounded streams, daemon/service loops, distributed execution, or async-runtime assumptions without architecture review.”
• Bounded memory is load-bearing. The arbitrator’s 512 MiB default and the spill machinery are a contract, not an implementation detail.
• One YAML chokepoint. All YAML parsing goes through clinker_plan::yaml, the only place that calls the underlying parser, where input limits are enforced.
• Path trust uses proof tokens: APIs that need a trusted path take a ValidatedPath, not a raw PathBuf.

This is why you check the invariants first. They’re the fixed points your change has to land between.

The crate layering is the boundary you’ll meet most

The most frequently-touched boundary is the dependency direction between crates. Dependencies flow one way, “from lower-level vocabulary toward applications”:

clinker ·20_CRATE_MAP.md ·from lower-level vocabulary toward applications doc @19acdcb4

clinker-core-types          (leaf vocabulary — no exec/config/schema types)
clinker-record
  → cxl
  → clinker-format
  → clinker-plan            (parses, validates, lowers → CompiledPlan)
  → clinker-exec            (consumes CompiledPlan, runs operators)
  → clinker-net
  → clinker / cxl-cli       (the application edges)

The rule that polices it:

clinker ·30_DESIGN_RULES.md ·Preserve the current crate layering. doc @19acdcb4

Preserve the current crate layering. clinker-plan parses config, resolves schemas, compiles CXL, validates, and produces a typed ExecutionPlanDag; it does not depend on runtime operators. clinker-exec consumes compiled plans and owns runtime dispatch.

You can prove this boundary holds by reading a manifest. clinker-plan’s dependencies list its lower layers, and pointedly not clinker-exec:

clinker-plan ·Cargo.toml ·clinker-format doc @19acdcb4

[dependencies]
clinker-core-types = { workspace = true }
clinker-record     = { workspace = true }
cxl                = { workspace = true }
clinker-format     = { workspace = true }
# clinker-exec is absent — plan must not depend on exec.

This is why an operator change splits the way it does: the config, plan, and lowering sites live in clinker-plan; the dispatch arm and operator body live in clinker-exec, which depends on clinker-plan, never the reverse. If your change tempted you to make clinker-plan call into a runtime operator, you’d have to add clinker-exec to that manifest, creating a cycle, and that’s the exact boundary the rule forbids. The manifest is the boundary made mechanical.

The engine’s three safe seams are a format (FormatReader/FormatWriter), a CXL builtin, and a transform operator. These are precisely the places where a new change stays inside one layer. A seam is the opposite of a boundary break: it’s where the architecture invites your code in.

Some decisions aren’t yours to make: the Decision Gate

The hardest part of planning a change is recognizing when you’ve hit a question you shouldn’t answer unilaterally. Clinker’s workflow names this explicitly: a Decision Gate is opened (and resolved) before implementation, whenever a change depends on an unresolved choice.

clinker ·GITHUB_ISSUE_AGENT_WORKFLOW.md ·Decision Gate doc @19acdcb4

The gate triggers cover:

• new dependency or cargo-deny exception
• public API behavior
• data model, schema, storage, or migration behavior
• auth, security, privacy, or credentials
• performance or bounded-memory tradeoff with unclear priority
• backward compatibility or breaking changes
• cross-crate or cross-service architecture boundaries
• any behavior the agent cannot validate from existing docs, tests, or source

That last bullet is the catch-all and the most important: if you can’t ground the right answer in existing source, tests, or docs, stop and gate it. Don’t guess and encode the guess. Resolving a gate means recording the chosen option, the rejected options, the source evidence, and the consequences. And if the question can’t be resolved, it’s written down rather than silently decided:

clinker ·80_OPEN_QUESTIONS.md ·Open Questions doc @19acdcb4

The open-questions register is where unresolved boundary questions live, things like “what is the intended boundary between clinker-schema and clinker-plan?” A contributor’s job is to recognize their change is about to answer one of these, and route it to a gate instead of deciding it in a pull request.

Mapping the names

“Decision Gate,” “Readiness Review,” and the “Grounding Result” are the project’s real terms. What this curriculum has called the “invariant register” maps onto the project’s “Architectural Rules” (30_DESIGN_RULES.md) and “Architectural Invariants” (10_ARCHITECTURE.md). Read those headings when you go looking.

Scoping a change: Use → Modify → Create

Change scoping is a repeatable three-question method you run on a change before writing code. Each question reads a real artifact you already have:

1. Which boundary does it touch? Read the crate map and the invariant list. Does the change keep dependencies flowing one way? Does it leave compile-before-execute, bounded memory, the YAML chokepoint, and path proof tokens intact?
2. What’s the blast radius? List every crate and behavior the change reaches. A change that stays inside one seam has a small radius; one that crosses a crate boundary or changes a public type has a large one.
3. Does any gate trigger fire? Hold the change up against the eight triggers. New dependency? Public API or schema? Cross-crate boundary? Anything you can’t ground in existing source? If yes, it’s an escalation, not a direct edit.

The three rungs below walk that method on a real change, then a variant, then one you scope from scratch.

Use: walk the method on a concrete change

Take a concrete proposal: “add a --max-rows N CLI flag that aborts a run after N records.” Walk the three questions, reading the real artifacts.

• Boundary. The flag is parsed at the application edge (clinker / cxl-cli) and the count is checked in the runtime as records flow, which is clinker-exec. Dependencies still flow one way: clinker → clinker-exec. No invariant is touched: the runtime stays finite and synchronous (a row cap, if anything, makes it more bounded), the YAML chokepoint is untouched, no path is involved.
• Blast radius. Two crates: argument parsing at the edge, a counter-and-abort check in the executor’s record loop. No new type crosses a crate boundary; nothing in clinker-plan changes.
• Gate triggers. It’s user-visible CLI behavior: public API behavior and arguably backward compatibility. A flag that changes how every run can terminate is a behavior decision, and you can’t fully ground “what should --max-rows 0 do?” or “is an early abort exit code 0 or non-zero?” in existing source. That fires a gate. Not because the code is hard, but because the behavior is a choice.

So the verdict for a tiny flag is: small blast radius, no invariant broken, and still a Decision Gate, because it changes public behavior in a way the source doesn’t already decide. Size of diff is not the signal; which boundaries and which triggers are.

Modify: change one thing and predict the new blast radius

Now vary the change. Instead of aborting after N records, suppose the flag is --sample-rate F: keep only a fraction F of records. One question changes its answer. Predict which, before revealing.

✋ Predict first

Re-scope --sample-rate F against the same three questions. Compared with --max-rows, does the boundary it touches change, does the blast radius grow, and does it fire a different (or additional) gate trigger? Write your answers before revealing.

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

• Boundary. Same edge (clinker parses the flag), but now the behavior lands inside a transform-shaped step in the record path: still clinker-exec, still one-way deps. The boundary touched is the same one.
• Blast radius. Slightly larger and sharper: dropping records is exactly the kind of thing a downstream count or a DLQ total will notice. The radius now includes observable output semantics (row counts, determinism), not just “the run ends early.” If the fraction is random, you’ve also touched determinism: does the same input produce the same output twice? That’s a property the engine’s testing strategy leans on.
• Gate triggers. Still public API behavior, but now data model / output behavior is squarely in play, and “is sampling deterministic, and seeded how?” is a behavior you can’t ground in existing source, so the catch-all trigger fires hard. The gate question is bigger than it was for --max-rows.

The lesson of the variant: changing one word in the proposal (cap → sample) moved the blast radius from “termination” to “output semantics and determinism,” and made the gate question weightier. That’s why you scope on paper first. The second-order effects are where careful contributors earn their title.

Create: scope a change from scratch, then verify

Your turn, with no walkthrough. Pick the change: “add a new parquet input format so pipelines can read .parquet files.” Scope it yourself with the three questions, write down a boundary, blast-radius, and gate verdict, then open the reveal to check yourself.

💡 Hint 1

Start with the crate map. A format is one of the engine’s three named seams. Which crate do FormatReader/FormatWriter implementations live in, and does adding one stay inside that layer? Then ask the dependency question: does a Parquet reader pull in a new third-party crate?

Scope it, then check yourself

• Boundary. A new format plugs into the FormatReader/FormatWriter seam in clinker-format, a contained extension point, low in the crate map. The format seam is designed for exactly this, so the layering stays intact: nothing above clinker-format needs to learn a new boundary.
• Blast radius. Mostly one crate (clinker-format) plus a registration point, except Parquet is a binary columnar format, so a real implementation almost certainly pulls in a new dependency (an arrow/parquet crate). That dependency, not the reader code, is the largest part of the radius: it brings transitive crates, a license, and a compile-time cost.
• Gate triggers. Two fire, clearly. New dependency or cargo-deny exception: a new third-party crate must be justified and pass the license audit. And bounded-memory tradeoff, since columnar readers can buffer whole row-groups, which collides head-on with the bounded-memory invariant; “how does a Parquet reader honor the memory cap?” is a behavior you can’t ground in existing source.

Verdict: even though it lands on a friendly seam with a small code footprint, the new dependency and the bounded-memory question make this a Decision Gate, not a direct PR. The seam keeps the code local; it does not exempt you from the dependency and memory decisions the dependency drags in.

To verify the layering claim against the real tree (in the clinker checkout): the format seam crate must not depend upward on plan or exec, and clinker-plan must still not depend on clinker-exec:

# the format seam sits below plan/exec — it depends on neither
grep -E 'clinker-(plan|exec)' crates/clinker-format/Cargo.toml \
  || echo 'clinker-format depends on neither plan nor exec — seam is low in the layering'
# and the plan→exec boundary still holds
grep -n 'clinker-exec' crates/clinker-plan/Cargo.toml \
  || echo 'clinker-exec absent from clinker-plan deps — layering holds'

Reading a change against the boundaries

// quick check

Your change makes the aggregation operator faster by having clinker-plan precompute a lookup the operator reads. To do it cleanly you'd add clinker-exec to clinker-plan's dependencies. What's the right move?

• Add the dependency; it's the most direct way to share the code
• Stop: this crosses the crate-layering boundary (plan must not depend on exec) and would create a cycle. It's a cross-crate architecture decision, so open a Decision Gate, or restructure so the shared type lives in a lower layer both can depend on
• Copy the operator code into clinker-plan to avoid the dependency

Ready for the capstones? Skip ahead →

Retrieval checkpoint

✓ Checkpoint: plan a change without breaking a boundary

💡 Hint 1

Read clinker-plan/Cargo.toml’s [dependencies] and confirm clinker-exec is only a comment, never a real dependency. Then run a change through the three scoping questions: which boundary, how big the blast radius, which gate triggers fire.

💡 Hint 2

Take a concrete idea, say “add a --max-rows CLI flag that aborts a run early.” Walk it: does it touch the layering (which crates)? Is it a public-API or user-visible behavior change (gate trigger)? Can you ground the intended behavior in existing source? The answer to “do I gate this?” falls out of those three questions.

What the lesson establishes

The engine’s invariants are written down (30_DESIGN_RULES.md, 10_ARCHITECTURE.md): compile-before-execute, finite/synchronous runtime, bounded memory, one YAML chokepoint, path proof tokens, and the one-way crate layering. The layering is the boundary you meet most, and a manifest proves it: clinker-plan does not depend on clinker-exec, so the plan/runtime split can’t be violated by accident. You scope a change before writing it by tracing its blast radius (which crates and boundaries) and holding it against the gate triggers. If it depends on an unresolved choice (new dependency, public API, schema, security, cross-crate architecture, or anything you can’t ground in existing source), you open a Decision Gate and resolve it first, recording evidence, not a guess, or capture it in the open-questions register.

// quick check

What is the catch-all condition that should trigger a Decision Gate even if no other trigger obviously applies?

• Any change that adds more than 100 lines
• Any behavior the agent cannot validate from existing docs, tests, or source; if you can't ground the right answer, you escalate rather than encode a guess
• Any change to a test file

// quick check

You propose adding a new parquet input format. The code touches one crate (clinker-format) on a friendly seam. Does the small code footprint mean no Decision Gate is needed?

• Yes: a change confined to one seam crate is always a direct PR
• No. A real Parquet reader pulls in a new third-party crate (a 'new dependency' trigger) and raises a bounded-memory question (columnar buffering), and both are gate triggers regardless of how little code you write
• No, but only because every new format requires a gate by rule

// quick check

MISCONCEPTION check: which statement about scoping a change is wrong and needs correcting?

• A change's blast radius can be larger than the files it edits; a crossed crate boundary or a new dependency reaches further than the diff
• If a change compiles and the diff is small, it's safe to land directly without checking the gate triggers
• Some safe changes still need a Decision Gate because they change public behavior the source doesn't already decide

// quick check

Spaced recall from the plan/runtime-boundary lesson: why is 'make clinker-plan call a clinker-exec operator' not merely discouraged but actually impossible to build, and how does that connect to the plan/runtime boundary?

• It compiles but panics at run time when the operator is missing
• Adding clinker-exec to clinker-plan's dependencies creates a dependency cycle Cargo refuses to build; the same plan/runtime boundary that makes the executor accept only a CompiledPlan (never raw config) is enforced one layer up by the crate graph
• It's only discouraged by the review checklist; nothing mechanical prevents it

You should be able to:

• You can name three architectural invariants and where they're written down
• You can read a crate's Cargo.toml and say whether a proposed dependency respects the layering
• You can trace a change's blast radius (the crates and boundaries it touches) before writing code
• You can list the Decision Gate triggers and identify, for a given change, whether one is needed

Verify in the checkout:

grep -n 'Preserve the current crate layering' docs/ai/30_DESIGN_RULES.md
grep -n 'clinker-exec' crates/clinker-plan/Cargo.toml || echo 'clinker-exec absent from clinker-plan deps — layering holds'
sed -n '/## Routing Rules/,/## Stop Conditions/p' docs/ai/GITHUB_ISSUE_AGENT_WORKFLOW.md
grep -cE '^### [0-9]+\.' docs/ai/80_OPEN_QUESTIONS.md

Where this leaves you: the capstones

You’ve read the engine end to end, extended each of its seams, and learned to scope a change and land it without breaking a boundary. Two capstones put it together:

• C1, synthetic: add a small extension end-to-end with tests, a new format (FormatReader/FormatWriter) or a CXL builtin. It’s stable across revisions and reviewable against a fixed rubric, the safe seams you now know.
• C2, real: a genuine, potentially-mergeable contribution selected fresh from the live backlog through the project’s Readiness-Review / Decision-Gate workflow: scoping, review conventions, and architectural reasoning under supervision.

That’s the whole arc of the Engine Track: from “build and run Clinker” to “make a real, boundary-respecting contribution to it.”

Prerequisites

• Review & the gauntlet

→

Leads to

• Engine Track — overview

Go deeper on the Rust (optional, one-directional; the typed handle behind the plan/runtime boundary this lesson polices is built on the same Result-returning compile step the engine’s error-handling lesson covers from first principles):

• Want the engine's error type and the gate between recoverable and fatal, from first principles? Skip ahead →

Glossary terms used: invariant, seam, blast radius, change scoping.

Mark this lesson complete