Add a CXL builtin
CXL is the small expression language you met in CXL: a staged language: emit full_name = first_name + " " + last_name,
compiled once and evaluated per record. Its string and numeric methods
(upper(), trim(), length(), …) are builtins. Adding one is the smallest real
contribution you can make to the engine, and it’s the perfect lesson in a subtle
architectural fact: a builtin lives in two separate tables, keyed by the same method
name, and nothing in the type system forces you to update both.
What you’ll be able to do
Section titled “What you’ll be able to do”- Name the two places a builtin is defined (the signature registry and the eval
match) and which compiler stage reads each. - Write the
BuiltinDefsignature entry and thedispatch_methodarm for a new scalar method. - Distinguish the two half-add failure modes: a signature with no eval arm versus an eval arm with no signature.
- Explain why a parse-to-eval test, not the compiler, is what proves you added a builtin rather than half of one.
New terms in this lesson (each is also expanded inline at the point you first need it):
- CXL: Clinker’s expression language; here, the surface you’re extending by one method.
- builtin: a method (
upper(),length(), …) callable on a CXL value, defined by a signature the typechecker reads and an implementation the evaluator runs. - signature table: the
BuiltinRegistryofBuiltinDefrecords, consulted at typecheck to learn a method’s receiver, args, and return type. - eval
match(the impl table): thedispatch_methodfunction, amatchon the method-name string, consulted at eval to actually compute the result. - the two-table seam: the un-checked join between those two tables: they share only the method-name string, so the compiler never makes you keep them in sync.
Value: the cell type a builtin receives and returns (from the Value cell).
Predict first
Section titled “Predict first”A builtin is a signature plus an implementation, stored apart
Section titled “A builtin is a signature plus an implementation, stored apart”Recall CXL’s staged pipeline: parse → resolve → typecheck → eval (CXL: a staged language). A builtin has to show up in two of those stages, and clinker stores those two halves in two different files:
- The signature, consulted at typecheck: what type does
s.upper()return? It lives in a registry ofBuiltinDefrecords. - The implementation, run at eval: what does
upper()actually do to the value? It lives in a bigmatchon the method name.
Start with the signature side. The registry is not a list of function pointers and not a
trait; it’s two hash maps of BuiltinDef records, built once:
cxl ·builtins.rs ·BuiltinDef type @19acdcb4
pub struct BuiltinDef { pub name: &'static str, pub receiver: TypeTag, // the type the method is called on pub args: Vec<TypeTag>, // expected argument types pub min_args: usize, pub max_args: Option<usize>, pub return_type: TypeTag, // what typecheck records for the call pub category: Category,} cxl ·builtins.rs ·BuiltinRegistry type @19acdcb4
/// Registry of all built-in methods and window functions.pub struct BuiltinRegistry { methods: ahash::HashMap<&'static str, BuiltinDef>, window_fns: ahash::HashMap<&'static str, BuiltinDef>,}BuiltinRegistry::new() fills methods imperatively. String methods are declared
through a little closure s(...) and an array, so each entry is a single readable line:
upper takes no args and returns a String:
let s = |name, args, min, max, ret| (name, BuiltinDef { name, receiver: TypeTag::String, args, min_args: min, max_args: max, return_type: ret, category: Category::String,});for (n, d) in [ s("upper", vec![], 0, Some(0), TypeTag::String), s("lower", vec![], 0, Some(0), TypeTag::String), // ...24 string methods in all] { methods.insert(n, d); }Notice what BuiltinDef does not have: any field holding the implementation. The
registry knows a method’s shape, never its behavior. That is the
signature table, the half the typechecker reads.
The implementation lives in a separate match
Section titled “The implementation lives in a separate match”The behavior is in a different file, the eval kernel’s dispatch_method, the
eval match: a match on the method-name string that returns Ok(None) for anything
it doesn’t recognize:
cxl ·builtins_impl.rs ·dispatch_method fn @19acdcb4
/// Dispatch a method call on a receiver value./// Returns None if the method is not a known built-in (caller should error).pub fn dispatch_method( receiver: &Value, method: &str, args: &[Value], regex: Option<&Regex>, span: Span, ctx: &EvalContext<'_>,) -> Result<Option<Value>, EvalError> { // ...null propagation first... match method { "upper" => Ok(Some(string_op(receiver, span, |s| { Value::String(s.to_uppercase().into()) }))), "lower" => Ok(Some(string_op(receiver, span, |s| { /* ... */ }))), // ...one arm per builtin... _ => Ok(None), // unknown method → caller raises an error }}So "upper" appears twice, in two files: once as a BuiltinDef in builtins.rs
(its return type is String), and once as a match arm in builtins_impl.rs (it
uppercases). The two are linked only by the string "upper": there is no shared enum,
no trait, nothing the compiler checks. That un-checked join is the two-table seam.
Why this matters: the silent half-add
Section titled “Why this matters: the silent half-add”Because the two tables are independent, you can update one and forget the other, and the program still compiles. The failure shows up at runtime or as a missing type:
- Signature only, no eval arm: typecheck is happy (it found the return type), but
evaluation hits the
_ => Ok(None)fall-through and the caller raises “unknown method.” - Eval arm only, no signature: evaluation works, but typecheck never finds a
BuiltinDef, so the call’s type silently falls back toAny(for scalar methods, clinker’s typechecker reads onlyreturn_typefrom the registry and does not even enforce arg types, so a missing signature degrades quietly rather than erroring).
Here is that two-table coupling as a runnable toy. The signature table and the impl match are keyed by the same string; comment out one half and watch a method become half-defined:
> output appears here — press Run
The lesson the toy makes concrete: adding a builtin is a two-file change, and the checklist for “did I really add it?” is human discipline plus a test, not the compiler.
The change-set, and the test that proves it
Section titled “The change-set, and the test that proves it”To add a scalar method end-to-end you edit two files:
crates/cxl/src/builtins.rs: add one entry toBuiltinRegistry::new()(pick the right category helper / array) so typecheck knows the receiver, args, and return type.crates/cxl/src/eval/builtins_impl.rs: add onematcharm todispatch_methodreturningOk(Some(value)), reusing helpers likestring_op.
(A closure-bearing method like map/filter, or a window function, needs extra
wiring in eval/compiled.rs and the window_fns table, which is out of scope here; the scalar
case is the clean first contribution.) Then prove it through the full stack, with a test that
parses, resolves, typechecks, and evaluates a program that calls the method:
cxl ·tests.rs ·string_methods test @19acdcb4
#[test]fn string_methods() { // drives whole programs like emit out = s.upper() through // parse -> resolve -> typecheck -> compile -> eval, asserting // s.upper() on "abc" yields Value::String("ABC"), etc.}Add a builtin: worked → completion → faded
Section titled “Add a builtin: worked → completion → faded”Now you write the change yourself, scaffolded from a fully-shown builtin, down to one you
add from a skeleton. To keep every rung runnable on std alone, the playgrounds model the
two-table mechanism the same way the toy above did: a signatures map standing in for the
BuiltinRegistry, and a dispatch match standing in for dispatch_method. The one idea
each rung adds is which of the two halves you’re editing, the same closed-match
discipline you already command from the Value cell, so the scaffolding fades fast.
Worked: a fully-shown builtin, length
Section titled “Worked: a fully-shown builtin, length”Here is length added end to end across both halves: an entry in signatures (the
typecheck side) and an arm in dispatch (the eval side). Run it and watch a single method
be fully defined: type known, value computed:
> output appears here — press Run
length is defined exactly twice: once in signatures so typecheck reports Int, once in
dispatch so eval returns the character count. Drop either line and the method is only
half-defined: the failure mode from the section above.
Completion: supply the eval half of trim
Section titled “Completion: supply the eval half of trim”Below, trim’s signature is already in place (signatures maps it to String), so
typecheck knows the return type. The eval half is missing: dispatch has a TODO
where its arm belongs. Add the one arm so s.trim() actually strips surrounding whitespace.
Without it you reproduce the signature-only half-add: the type prints String but eval
says “unknown method.”
> output appears here — press Run
💡 Hint 1
"upper" arm: a "trim" => Some(...) line. Rust’s str::trim returns a &str; turn it
into an owned String with .to_string() so the arm’s type matches.Show solution
"trim" => Some(recv.trim().to_string()),With both halves present, trim is a real builtin: typecheck reports String from the
signature table, and eval returns the trimmed value from the match. Comment the arm back out
and you’ve recreated the exact signature-only failure: type known, eval unknown.
Faded: add reverse yourself, both halves
Section titled “Faded: add reverse yourself, both halves”Now with much less scaffolding. Add a brand-new builtin, reverse, that returns the
characters of the receiver in reverse order. This is the real two-file discipline in
miniature: you must touch both tables (a signatures entry and a dispatch arm), or
the method is only half-defined. The skeleton has both halves missing; fill them so
s.reverse() on "abc" prints "cba" with type String.
> output appears here — press Run
💡 Hint 1
m.insert("reverse", "String");. Eval:
a "reverse" => Some(...) arm. Reverse a string by characters with
recv.chars().rev().collect::<String>(). Leave out either edit and main prints Any or
“unknown method”: the half-add made visible.Show solution
// signature half, in signatures():m.insert("reverse", "String");
// eval half, in dispatch's match:"reverse" => Some(recv.chars().rev().collect::<String>()),Mapped onto the real crate, those are exactly the two edits: the signatures insert is the
s("reverse", vec![], 0, Some(0), TypeTag::String) entry in BuiltinRegistry::new()
(builtins.rs), and the dispatch arm is
"reverse" => Ok(Some(string_op(receiver, span, |s| Value::String(s.chars().rev().collect::<String>().into()))))
in dispatch_method (builtins_impl.rs). Same two-table shape, just with the real types.
The last step is the proof: copy a case from string_methods and assert s.reverse() on
"abc" evaluates to "cba". Driving it parse → typecheck → eval is what confirms you added
the whole builtin, not half of one.
Real-source grounding
Section titled “Real-source grounding”The two playground stand-ins map directly onto the cited crate. The signatures map is the
BuiltinRegistry, built once, read at typecheck for a method’s return_type:
cxl ·builtins.rs ·BuiltinRegistry type @19acdcb4
// signature half: typecheck looks up the BuiltinDef and reads return_typelet bd = registry.methods.get("reverse"); // Some(BuiltinDef) → return_type = String// // None → call's type falls back to AnyThe dispatch match is dispatch_method, read at eval, with the catch-all that turns a
missing arm into a run-time error:
cxl ·builtins_impl.rs ·dispatch_method fn @19acdcb4
// eval half: a missing arm falls through to the catch-allmatch method { "reverse" => Ok(Some(/* reversed value */)), // ... _ => Ok(None), // None here → caller raises "unknown method" at run time}What the compiler enforces: nothing across the seam. Each table compiles on its own; the
_ => Ok(None) arm means even a method with no eval entry type-checks the match. There is
no path by which forgetting one table becomes a build error.
What a junior might misread: assuming the registry and the match are kept in sync
automatically (they’re not, since only the method-name string joins them), or reading _ => Ok(None)
as “returns null” rather than “signal the caller to raise unknown method.” Both mistakes
produce a builtin that looks added but isn’t.
Why-bridge: why two tables, and why no enum links them
Section titled “Why-bridge: why two tables, and why no enum links them”Why split a builtin’s shape from its behavior across two files at all? Because the two halves
are read at different stages, for different reasons, and CXL keeps stage concerns apart
(CXL: a staged language). Typecheck runs once, at plan time, and needs only the shape (receiver,
args, return type) to reject a_string.length() > a_date before record one. Eval runs per
record and needs only the behavior. Bolting the implementation onto BuiltinDef would
drag eval-time closures into the plan-time registry; keeping them apart lets typecheck stay a
pure shape check.
The cost of that separation is the seam: nothing forces the two tables to agree. Clinker
could have made one enum Builtin { Upper, Lower, … } that both stages match on
exhaustively, and then forgetting a half would be a compile error, the way adding a node variant
is in dispatching a node. It doesn’t: the registry is string-keyed hash maps, chosen so methods can be
declared in compact data-driven arrays rather than a giant enum plus two mirrored matches. The
trade is real (data-driven brevity in exchange for a compiler that won’t catch a half-add), so
the discipline moves to a test.
// quick check
You add a BuiltinDef for a new method `slugify` to BuiltinRegistry::new() but forget to add an arm to dispatch_method. What happens when a pipeline calls s.slugify()?
The two tables are linked only by the method-name string, with no compile-time check between them. A signature without an eval arm typechecks fine, then fails at eval via the catch-all. That gap is exactly why a parse-to-eval test (like string_methods) is the real proof you added a builtin, not just half of one.
Retrieval checkpoint
Section titled “Retrieval checkpoint”Connections
Section titled “Connections”You’ve extended the expression language by one method and felt the real shape of the change: two tables, one string key, no compiler between them, proven by a parse-to-eval test. The deeper pattern, typecheck once at plan time, eval per record, is the staged-language model from CXL: a staged language, now seen from the contributor’s side. Next: extend the engine’s edges instead of its language by adding a whole new file format behind the reader/writer seam.
Go deeper on the Rust (optional, one-directional, for the dispatch idea from first principles in The Rust Book):
Glossary terms used: CXL, Value.