Sources & schema — name your columns

The question for this lesson: before Clinker can transform your data, it has to read it and know what each column is: text, a number, a date. Where does that come from?

What you’ll be able to do

• Declare a schema on a source: name and type each column.
• Choose the right type for a column, and know which types exist.
• Decide what happens to columns that aren’t in your schema.

Runtime types come from the schema

✋ Predict first

A CSV has a column `amount` with values like 4000 and 1200. If you don't tell Clinker that `amount` is a number, how should it treat the value 4000?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

As text: the characters “4000”, not the number 4000. Runtime ingestion does not infer types from the file: declared cells are parsed to their declared types. Undeclared CSV cells remain text when the auto_widen policy retains them. That’s why you declare a schema: so amount can be a number you can compare and do math on.

This is the most important thing to know about sources: you declare the columns and their types; runtime ingestion enforces them. A CSV header only names columns; it never sets types.

Declaring a schema

A source’s schema: is a list of { name, type } entries:

- type: source
  name: customers
  config:
    type: csv
    path: ./data/customers.csv
    options: { has_header: true }
    schema:
      - { name: customer_id, type: string }
      - { name: status,      type: string }
      - { name: amount,      type: int }
      - { name: signed_up,   type: date }

The types you can use in an inline schema:

Type	For
string	text
int	whole numbers
float	approximate binary floating-point numbers
decimal	exact base-10 numbers; declare precision and scale for money
bool	true / false
date, date_time	a date, or a date + time
array, map	a list, or a set of key/values

🌱 New here? — picking a type

Match the type to how you’ll use the column, not just what it looks like. A zip code looks like a number but you never do math on it, so make it string and the leading zero (“01970”) survives. An integer count should be int; for money, use decimal with an explicit precision and scale.

The date-and-time spelling is date_time. Exact decimal columns are supported:

schema:
  - { name: amount, type: decimal, precision: 12, scale: 2 }
  - { name: signed_up, type: date, format: "%Y-%m-%d" }

A CSV decoder first produces text. Source admission then parses that text against these declarations before downstream nodes see it. Declare amount as int and CXL receives an integer without .to_int(). Declare it as string and it stays text until you explicitly convert it. Invalid input rejects the row under the run’s error policy; it never silently falls back to a string.

clinker guess is a separate authoring tool for explicitly marked numeric columns. Its default preview is read-only; it proposes int or float. Resolve those placeholders before running a pipeline. It does not guess an entire schema or change runtime types behind your back.

Only the listed types are real

“Declared, not inferred” cuts both ways: the type you write has to be one Clinker knows. A made-up type like dollars is not silently accepted or guessed. It’s rejected before any data is read. If you change amount’s type to dollars and validate:

schema:
  - { name: customer_id, type: string }
  - { name: amount, type: dollars }    # not a real type

clinker run pipeline.yaml --dry-run rejects the unknown type before reading data. Use one of the supported types above. numeric is an inference-only placeholder, not an executable source type; nullable wraps a supported concrete type.

// quick check

A column holds US zip codes like 01970. What type should it be?

• string, because you never do math on it and you must keep the leading zero
• int, because it looks like a number
• float, to be safe

Columns that don’t fit: on_unmapped

What about columns in the file that you didn’t list in the schema? The on_unmapped setting decides:

• auto_widen (default): retain undeclared CSV cells in the engine sidecar. A Sink with include_unmapped: true can include them in its output; they do not become declared CXL columns.
• reject: fail if the file has a column you didn’t declare.
• drop: silently ignore them.

config:
  type: csv
  path: ./data/customers.csv
  options: { has_header: true }
  on_unmapped: reject     # be strict: an unexpected column is an error
  schema:
    - { name: status, type: string }

Try it

Predict, then run: this source declares amount as int. Trace what the output type of amount is, versus customer_id (declared string).

✋ Predict first

amount is declared int and customer_id is declared string. After the source reads a row with customer_id=C001 and amount=4000, what type is each value?

Reveal the expected result

=== Resolved Outputs ===

'out' → ./output/customers.csv
    [if_exists=overwrite, write_meta=false]

Dry run: plan valid, 1 inputs, 1 outputs, 0 transforms

pipeline.yaml // editable

reset ⌖ explain ✓ dry-run ▶ run

pipeline: name: customer_schema nodes: - type: source name: customers config: name: customers type: csv path: ./data/customer-schema.csv options: { has_header: true } schema: - { name: customer_id, type: string } - { name: amount, type: int } - type: sink name: out input: customers config: name: out type: csv path: ./output/customers.csv

A --dry-run validates the config and the declared schema without reading any data, so the summary line reports the shape of the pipeline (here, one source, one output, and no transforms) and exits cleanly. It does not print a column-by-column type table; the place the types live is the schema: you wrote.

✓ Checkpoint

// quick check

Where do a column's types come from?

• The schema you declare on the source, where Clinker never infers them
• Clinker guesses from the first few rows of the file
• The output node decides them

// quick check

Looking back at the notional machine from lesson 00 (source → nodes → output, with a DLQ off to the side): where does the source sit, and what does it hand to the rest of the pipeline?

• At the front, reading the file and producing records that flow into the downstream nodes
• At the end, writing the final output file
• Off to the side, only catching rejected rows

Common mistakes

• Leaving a number as text: if you’ll compare or total a column, declare it int/float.
• Using datetime: the YAML spelling is date_time.
• Casting every CSV field: declared numeric columns are already typed before CXL runs.

You should be able to:

• Declare a schema with the right type for each column
• Explain why a zip code should be string, not int
• Say what auto_widen, reject, and drop do to undeclared columns

Verify in the checkout:

clinker run customer_etl.yaml --dry-run
clinker run customer_etl.yaml --explain