Skip to content

Add a CXL builtin

CXL is the small expression language you met in CXL: a staged language: emit full_name = first_name + " " + last_name, compiled once and evaluated per record. Its string and numeric methods (upper(), trim(), length(), …) are builtins. Adding one is the smallest real contribution you can make to the engine, and it’s the perfect lesson in a subtle architectural fact: a builtin lives in two separate tables, keyed by the same method name, and nothing in the type system forces you to update both.

  • Name the two places a builtin is defined (the signature registry and the eval match) and which compiler stage reads each.
  • Write the BuiltinDef signature entry and the dispatch_method arm for a new scalar method.
  • Distinguish the two half-add failure modes: a signature with no eval arm versus an eval arm with no signature.
  • Explain why a parse-to-eval test, not the compiler, is what proves you added a builtin rather than half of one.

New terms in this lesson (each is also expanded inline at the point you first need it):

  • CXL: Clinker’s expression language; here, the surface you’re extending by one method.
  • builtin: a method (upper(), length(), …) callable on a CXL value, defined by a signature the typechecker reads and an implementation the evaluator runs.
  • signature table: the BuiltinRegistry of BuiltinDef records, consulted at typecheck to learn a method’s receiver, args, and return type.
  • eval match (the impl table): the dispatch_method function, a match on the method-name string, consulted at eval to actually compute the result.
  • the two-table seam: the un-checked join between those two tables: they share only the method-name string, so the compiler never makes you keep them in sync.
  • Value: the cell type a builtin receives and returns (from the Value cell).

A builtin is a signature plus an implementation, stored apart

Section titled “A builtin is a signature plus an implementation, stored apart”

Recall CXL’s staged pipeline: parse → resolve → typecheck → eval (CXL: a staged language). A builtin has to show up in two of those stages, and clinker stores those two halves in two different files:

  1. The signature, consulted at typecheck: what type does s.upper() return? It lives in a registry of BuiltinDef records.
  2. The implementation, run at eval: what does upper() actually do to the value? It lives in a big match on the method name.

Start with the signature side. The registry is not a list of function pointers and not a trait; it’s two hash maps of BuiltinDef records, built once:

cxl ·builtins.rs ·BuiltinDef type @19acdcb4
pub struct BuiltinDef {
pub name: &'static str,
pub receiver: TypeTag, // the type the method is called on
pub args: Vec<TypeTag>, // expected argument types
pub min_args: usize,
pub max_args: Option<usize>,
pub return_type: TypeTag, // what typecheck records for the call
pub category: Category,
}
cxl ·builtins.rs ·BuiltinRegistry type @19acdcb4
/// Registry of all built-in methods and window functions.
pub struct BuiltinRegistry {
methods: ahash::HashMap<&'static str, BuiltinDef>,
window_fns: ahash::HashMap<&'static str, BuiltinDef>,
}

BuiltinRegistry::new() fills methods imperatively. String methods are declared through a little closure s(...) and an array, so each entry is a single readable line: upper takes no args and returns a String:

let s = |name, args, min, max, ret| (name, BuiltinDef {
name, receiver: TypeTag::String, args,
min_args: min, max_args: max, return_type: ret, category: Category::String,
});
for (n, d) in [
s("upper", vec![], 0, Some(0), TypeTag::String),
s("lower", vec![], 0, Some(0), TypeTag::String),
// ...24 string methods in all
] { methods.insert(n, d); }

Notice what BuiltinDef does not have: any field holding the implementation. The registry knows a method’s shape, never its behavior. That is the signature table, the half the typechecker reads.

The implementation lives in a separate match

Section titled “The implementation lives in a separate match”

The behavior is in a different file, the eval kernel’s dispatch_method, the eval match: a match on the method-name string that returns Ok(None) for anything it doesn’t recognize:

cxl ·builtins_impl.rs ·dispatch_method fn @19acdcb4
/// Dispatch a method call on a receiver value.
/// Returns None if the method is not a known built-in (caller should error).
pub fn dispatch_method(
receiver: &Value, method: &str, args: &[Value],
regex: Option<&Regex>, span: Span, ctx: &EvalContext<'_>,
) -> Result<Option<Value>, EvalError> {
// ...null propagation first...
match method {
"upper" => Ok(Some(string_op(receiver, span, |s| {
Value::String(s.to_uppercase().into())
}))),
"lower" => Ok(Some(string_op(receiver, span, |s| { /* ... */ }))),
// ...one arm per builtin...
_ => Ok(None), // unknown method → caller raises an error
}
}

So "upper" appears twice, in two files: once as a BuiltinDef in builtins.rs (its return type is String), and once as a match arm in builtins_impl.rs (it uppercases). The two are linked only by the string "upper": there is no shared enum, no trait, nothing the compiler checks. That un-checked join is the two-table seam.

Because the two tables are independent, you can update one and forget the other, and the program still compiles. The failure shows up at runtime or as a missing type:

  • Signature only, no eval arm: typecheck is happy (it found the return type), but evaluation hits the _ => Ok(None) fall-through and the caller raises “unknown method.”
  • Eval arm only, no signature: evaluation works, but typecheck never finds a BuiltinDef, so the call’s type silently falls back to Any (for scalar methods, clinker’s typechecker reads only return_type from the registry and does not even enforce arg types, so a missing signature degrades quietly rather than erroring).

Here is that two-table coupling as a runnable toy. The signature table and the impl match are keyed by the same string; comment out one half and watch a method become half-defined:

rust // editable

The lesson the toy makes concrete: adding a builtin is a two-file change, and the checklist for “did I really add it?” is human discipline plus a test, not the compiler.

The change-set, and the test that proves it

Section titled “The change-set, and the test that proves it”

To add a scalar method end-to-end you edit two files:

  1. crates/cxl/src/builtins.rs: add one entry to BuiltinRegistry::new() (pick the right category helper / array) so typecheck knows the receiver, args, and return type.
  2. crates/cxl/src/eval/builtins_impl.rs: add one match arm to dispatch_method returning Ok(Some(value)), reusing helpers like string_op.

(A closure-bearing method like map/filter, or a window function, needs extra wiring in eval/compiled.rs and the window_fns table, which is out of scope here; the scalar case is the clean first contribution.) Then prove it through the full stack, with a test that parses, resolves, typechecks, and evaluates a program that calls the method:

cxl ·tests.rs ·string_methods test @19acdcb4
#[test]
fn string_methods() {
// drives whole programs like emit out = s.upper() through
// parse -> resolve -> typecheck -> compile -> eval, asserting
// s.upper() on "abc" yields Value::String("ABC"), etc.
}

Add a builtin: worked → completion → faded

Section titled “Add a builtin: worked → completion → faded”

Now you write the change yourself, scaffolded from a fully-shown builtin, down to one you add from a skeleton. To keep every rung runnable on std alone, the playgrounds model the two-table mechanism the same way the toy above did: a signatures map standing in for the BuiltinRegistry, and a dispatch match standing in for dispatch_method. The one idea each rung adds is which of the two halves you’re editing, the same closed-match discipline you already command from the Value cell, so the scaffolding fades fast.

Here is length added end to end across both halves: an entry in signatures (the typecheck side) and an arm in dispatch (the eval side). Run it and watch a single method be fully defined: type known, value computed:

rust // editable

length is defined exactly twice: once in signatures so typecheck reports Int, once in dispatch so eval returns the character count. Drop either line and the method is only half-defined: the failure mode from the section above.

Below, trim’s signature is already in place (signatures maps it to String), so typecheck knows the return type. The eval half is missing: dispatch has a TODO where its arm belongs. Add the one arm so s.trim() actually strips surrounding whitespace. Without it you reproduce the signature-only half-add: the type prints String but eval says “unknown method.”

rust // editable
💡 Hint 1
The signature half is already done, so you only owe the eval arm. Mirror the "upper" arm: a "trim" => Some(...) line. Rust’s str::trim returns a &str; turn it into an owned String with .to_string() so the arm’s type matches.
Show solution
"trim" => Some(recv.trim().to_string()),

With both halves present, trim is a real builtin: typecheck reports String from the signature table, and eval returns the trimmed value from the match. Comment the arm back out and you’ve recreated the exact signature-only failure: type known, eval unknown.

Now with much less scaffolding. Add a brand-new builtin, reverse, that returns the characters of the receiver in reverse order. This is the real two-file discipline in miniature: you must touch both tables (a signatures entry and a dispatch arm), or the method is only half-defined. The skeleton has both halves missing; fill them so s.reverse() on "abc" prints "cba" with type String.

rust // editable
💡 Hint 1
Two edits, one per table. Signature: m.insert("reverse", "String");. Eval: a "reverse" => Some(...) arm. Reverse a string by characters with recv.chars().rev().collect::<String>(). Leave out either edit and main prints Any or “unknown method”: the half-add made visible.
Show solution
// signature half, in signatures():
m.insert("reverse", "String");
// eval half, in dispatch's match:
"reverse" => Some(recv.chars().rev().collect::<String>()),

Mapped onto the real crate, those are exactly the two edits: the signatures insert is the s("reverse", vec![], 0, Some(0), TypeTag::String) entry in BuiltinRegistry::new() (builtins.rs), and the dispatch arm is "reverse" => Ok(Some(string_op(receiver, span, |s| Value::String(s.chars().rev().collect::<String>().into())))) in dispatch_method (builtins_impl.rs). Same two-table shape, just with the real types. The last step is the proof: copy a case from string_methods and assert s.reverse() on "abc" evaluates to "cba". Driving it parse → typecheck → eval is what confirms you added the whole builtin, not half of one.

The two playground stand-ins map directly onto the cited crate. The signatures map is the BuiltinRegistry, built once, read at typecheck for a method’s return_type:

cxl ·builtins.rs ·BuiltinRegistry type @19acdcb4
// signature half: typecheck looks up the BuiltinDef and reads return_type
let bd = registry.methods.get("reverse"); // Some(BuiltinDef) → return_type = String
// // None → call's type falls back to Any

The dispatch match is dispatch_method, read at eval, with the catch-all that turns a missing arm into a run-time error:

cxl ·builtins_impl.rs ·dispatch_method fn @19acdcb4
// eval half: a missing arm falls through to the catch-all
match method {
"reverse" => Ok(Some(/* reversed value */)),
// ...
_ => Ok(None), // None here → caller raises "unknown method" at run time
}

What the compiler enforces: nothing across the seam. Each table compiles on its own; the _ => Ok(None) arm means even a method with no eval entry type-checks the match. There is no path by which forgetting one table becomes a build error.

What a junior might misread: assuming the registry and the match are kept in sync automatically (they’re not, since only the method-name string joins them), or reading _ => Ok(None) as “returns null” rather than “signal the caller to raise unknown method.” Both mistakes produce a builtin that looks added but isn’t.

Section titled “Why-bridge: why two tables, and why no enum links them”

Why split a builtin’s shape from its behavior across two files at all? Because the two halves are read at different stages, for different reasons, and CXL keeps stage concerns apart (CXL: a staged language). Typecheck runs once, at plan time, and needs only the shape (receiver, args, return type) to reject a_string.length() > a_date before record one. Eval runs per record and needs only the behavior. Bolting the implementation onto BuiltinDef would drag eval-time closures into the plan-time registry; keeping them apart lets typecheck stay a pure shape check.

The cost of that separation is the seam: nothing forces the two tables to agree. Clinker could have made one enum Builtin { Upper, Lower, … } that both stages match on exhaustively, and then forgetting a half would be a compile error, the way adding a node variant is in dispatching a node. It doesn’t: the registry is string-keyed hash maps, chosen so methods can be declared in compact data-driven arrays rather than a giant enum plus two mirrored matches. The trade is real (data-driven brevity in exchange for a compiler that won’t catch a half-add), so the discipline moves to a test.

// quick check

You add a BuiltinDef for a new method `slugify` to BuiltinRegistry::new() but forget to add an arm to dispatch_method. What happens when a pipeline calls s.slugify()?

You’ve extended the expression language by one method and felt the real shape of the change: two tables, one string key, no compiler between them, proven by a parse-to-eval test. The deeper pattern, typecheck once at plan time, eval per record, is the staged-language model from CXL: a staged language, now seen from the contributor’s side. Next: extend the engine’s edges instead of its language by adding a whole new file format behind the reader/writer seam.

Go deeper on the Rust (optional, one-directional, for the dispatch idea from first principles in The Rust Book):

Glossary terms used: CXL, Value.